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More and more citizens want to know In concrete terms what they are 
receiving in retmn for their hard-earned tax dollais invested in public 
education. Governors, legislators. State leaders, and blue-ribbon 
committees are increasiqgiy being called upon to provide more flnaixrial 
support for public schools. Ih^ naturally want to be assured that this 
support yields an adequate return. 

As a result of these demands, concern about student performance, school 
results, and accountability for outcomes and improvement continues to 
grow. This report attempts to provide altematives to citizens, 
businesspeople. parents. pohcym Jsas. and educators who expect increased 
school accountability and productivity. 

The public may be displeased at times by the less than enthusiastic 
response educators might give to initial cries for accoimtabillty in 
education. Educator displeasure is dlen not about the principle that 
schools should be held accountable. Rather it stems bom their l^itimate 
concern about the accountability measures selected These measures may 
focus education too nanowfy. ma^ cause schools to pay attention to rote 
learning anfy. may encourage teaching to test items, or may be 
inappropriately used. This report discusses such matters and offers options 
for providir^ school accountability in a way that could minimize a number 
of frequentty e3q>ressed concerns. 

Unfortunate^, or fortunately, from some points of view, there is no one. tried 
and tested solution to school accountability. There are. however, several 
principles upon which some agreement can be reached. Our Study Group, 
staff, and researchers reached agreement on these principles after careful 
study of accountability measures in the 10 States represented in the group; 
fieldworkat schools in 4 States; analysis of a 50-State surv^. review the 
experience of 5 State-based, national organizations; and research in this 
area. As a result, die background information, findings, and 
recommendations in this report represent informed Judgment from an 
unusual^ diverse mixture ctf sources and people working together. 

Good public education for all youth is critical to meet the societal and 
economic challenges facing America. A thoughtful approach to school 
accountability is one ingredient in providing that good education. 

The reader is encouraged to use the suggested accountability measures not 
with "an ax to grind" but as "a field to sow." 



Terry Peterson 

Chair, Office of EduccLtLonal Research and Improvement 
State AccouTtUibfllty Study Group 
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Preface 



As In business and government. In education, reliable infonnation is one 

to better perfonnance. This belief has guided mai^ recent efforts ctf the 
U.S. Department of Education* Ftom American fiducodon: Making It Work 
and WhatWorks: Research AboutTeachtig and Leam^ a host of other 
projects, products, and activities, better infonnation about the condition of 
schools has been made available to the public. 

While the Department has been woiidng at the national level. State 
education leaders across the countiy have been woildng to improve the 
caliber of schools and build enduring systems for collecting and reporting 
soimd information on their perfomiance. We knew that States like 
California. Illinois, and South Carolina were developing publL. "report 
cards* on the perfonnance of individual schools throughout their States. We 
knew that New Jers^. South Carolina, and Georgia were pkmeermg a 
concept of educational ^bankruptcy" for schools and school systems found 
to perform chronically at substandard levels. We knew that other States 
were devisir^ rewards and assistance to i; ^prove school performance and 
that policy was changing faster than practically ar^one could easily keep 
track of. 

Accordip^, in the summer of 1987, the Department convened a study 
^up of individuals bom 10 States and five national organizations, all of 
them knowledgeable about promising new trends in State educational 
accountability. Thdr charge was to help us understand these developments 
and. based on their collective expectance, to offer recommendations to other 
States. In less than one year's time th^ were able to do so. This book is the 
report of their work. 

This OERI State Accountability Study Group, as we called it. was made up 
of e3q)erienced. State-based people fitxm governors* offices. State legislatures 
(or their stafO^. departments of education, pxid local school districts. Tliey 
were Joined by staff with similar but national perspectives— from the 
Natbmal Governors" Association (NGA). the National Conference of State 
legislatures (NCSU. the Council of Chfef State School Officers (CCSSO). the 
National Association of State Boards (rf" Education (NASBE). and our own 
Department staff. Two policy am. ysts. Lorraine McDonneD and Jearmie 
Oakes of the RAND Corporation, as part of their woric with the Center for 
Research on Educational Policy (CFRE). were the Study Gnmp members 
who drafted th? report. Tlie group was chaired by Terry Peterson of South 
Carolina, currentfy executive director d the State's business- education 
oversight committee and previous^ Governor Dk:k Riley's chief education 
aide. 

Meeting in July and November 1987 and again in February 1988. the Study 
Group reported on the practices of their State accountabil^ systems, how 
they developed over time, and the problems and successes the policies 
encountered. After the first meeting, the CCSSO was commissioned to 
collect infonnation on current practk^es of all 50 States and the District of 
Coltmibia concerning k^ elements cS State educational accoimtability . This 
is the source of much of the 50-State data fotmd in this report. The exact 
numbers— how many States do what— have been changing rapidly, even 



durlpg the brief life uf the Study Group. Nonetheless, this report presents 
the best analysis and most current information available on these evolvlpg 
policies and practices. 

If we are successful this report win encourage its own obsolescence, as 
States move to refine the data th^ collect the audlerK:es to whom they 
report It the f^nnat and provision for Its interpretation, and most of all, the 
skin and wisdom with which the States use that information to Improve 
their schools. As such chaises occur, our purpose is to assist those who are 
bulldirigaccouritabilttyflysteixis by offerlr^ them an overview nature 
(tf the technical and polttJcal Issues Involved and the best advice firam 
experienced colleagues. 

The creation of sound State accountability systems is one of the most potent 
(tf the recent education reforms of the States. The systems will produce 
comprehensive Information about schools for years to come. At their besc. 
th^ win allow polkTmakers, educators, and the public to know how wdl 
their schools are vow dotng and how to hc^ them do better in the future. 
As this report makes clear, this is no easy task. In mar^ comnmnttles. it 
involves new definitions of who is going to be held accountable to whom for 
what. These are potentially fi:actknis Issues whose resolution bes in clarity 
of purpose, mutuality of trust and evolution of consensus on the point that 
solid Information and well wrought aocountabfltty mechanisms are f'^^^t*^^ 
for the success of schools. 



Chester E. Flm Jn 
Assistant Secretary 
and Coimsdor to the Secretary 
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Executive Summary 

The majortty of States pursue accountability In public education by 
collectiiig and reportizig data about schools— student attendance and test 
performance, resources spent, the types of programs offered, and the like. 
But the Idiids of data collected, arid how those data are reported and t 
in holdir^ schools accountable, differ from State to State. This report, 
generated by the OERI State Accountability Study Group, is designed for 
State poUcymakers who are interested in refining their existing educational 
accountability systems. The report raises basic design issues and provides 
guidelines based on research and the eq>erience of other States. However, it 
does not supply definitive answers. Each State must design an 
accountability strategy that supports its own educational goals and needs 
and is consistent with its own political traditions. 



Basic Elements 
of a Perfonnance 
Accountability 
System 



A performance accountability system is a set of indicators or statistics that 
provides information about how well schools are performlr^g. Data from the 
system should allow pobcymakcrB to compare performance over time, 
against standards, and with comparable educational entities tfbr example. 
States with other States, schools with other schools). By their choice of 
indicators, policymakers determine who wHl be held accountable. Jbr what 
and to whom 



State Perforaiance 
Accountability 
Systems: 
I>escription and 
Trends 



To produce Information that is reliable, fair, and useful for improving 
schools, the indicators in the accountability system should 



Measure the central features of schooling: 

Measure ii^iat is actual^ beirig taiight or corisidered iriiportarit for 

students to know; 

Provide information that is poUcy-relevant; 
Focus on the school site; 
Allow for fair comparison; and 

Maximize the usefulness of the data collected and minimize the burden 
of conectirglL 



An accountability system Is more effective and more likely to be used if it 
relies on a model that represents the entire schooling enterprise. 

All 50 States and the District of Columbia collect perfonnance data about 
schooling. Although the systems vary considerably, they can be classified 
along two mayor dimensions: 1) who Is primarily responsible for the system 
and 2) whether the system is linked with other State policies. The dominant 
pattern, found in 21 States. Is one In which the State has primary 
responsibility for deckling what data to collect and how to organize and 
report them and what State-level policy action, based on the data, will 
irriprove educatloa No State system can accurately be considered a *pure* 
type, however, and States with the same general type can have disshnllar 
systems. 



vli 



states must also specify: 



What the Indicators win be: 

• Student peifonnance data? 

• School resources and processes? 

• Backgroimd data on students and conimimlUes? 



The lercl at irtileh data will be collected: 

• State? 

• District? 

• School? 

• Classroom? 

• Student? 



The level at which data will be organized and reported: 

• State? 

• District? 

• School? 



The mechanlams for reporting: 

• Reports to the public and media? 

• Reix>rts to parents? 

• Data made available on request? 



How schools win be compared: 

• With an other schools? 

• With slinnar schools? 

• With their own past performance? 



Each of these decisions shapes the content and form of a State's 
accountabiUty system. Whether the resulting system is effective, feasible, 
and acceptable to practitioners and poUqrmakers depends on the State's 
poUtlcal context and existing poUcy precedents. Consequently, no best 
design can be prescribed for an States, and. not surpiislngly. no two States 
have made Identical decisions in aU areas. 



Uses and In designing their systems. State policymakers make certain assumptions 
Consequences of accountabmty data win be used— both for their own purposes 

Pi^rfrirmanr** ^ ^ districts and schools. However, the way the systems are 

renormance actuaUy used depends on how local educators choose to Incorporate them 

Accountability and the information they generate into school practice. 

Systems 
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Tlie investigation of the in \iiMch accountability systems are 
influencii^ schools show mixed results. Although these systems are 
powerful policy tools that have significantly changed schooHevel planning 
and teaching activities, mai^ respondents feel that they motivate teachers 
and administrators to focus narrowly on tests. 

State-controlled systems differ frorn locally controlled systems. With 
State-controlled accountability, \ . jl districts and schools appear to take 
the systems seriously, and have changed their beliavior in order to improve 
performance. However, that performaix^e has been narrowly defined by the 
focus and scope of State tests. Many school persoimel therefore do not 
consider the systems useful or valid measures of their schools' performance. 
Instead, th^ respond to the systems because they feel pressure for their 
students to do well. Althou^ local superintendents and principals 
recognize the need to use other indicators as well in Jud^ng a district's or 
school's performance, th^ see the State as being primarily concerned with 
test scores; consequently, th^ have charmeled much of their energy to 
improving performance on that one indicator. 

Locally controlled systems have both more and less effect than 
State-controlled ^stems. In disblcts where school officials are committed to 
using aocountabiUty data to improve schools, the locally controlled system 
influences school practice more than a State-operated one. However, 
whether this process occurs or not depends cirtlrely on local priorities and 
capacity. Some districts use the data extensively to inform practice, while 
others do little more than meet State testing requirements. 

Unresolved Policymakers face a nmriber of major dilenmias in developing or modifying 
Dilemmas ^ state accountablltty system, including: 

Balandng ovenlght and improvement. 

Policymakers can ensure that an accotmtability ^stem provides useful 
information for improving schools, not Just monitoring them, by carefully 
choosing the types of data that are collected and reported and t)y selecting 
an appropriate mix of rewards, sanctions, and technical assistance. States 
need to experiment, then modify their systems in light of experience. 

Deteminlng the a^ro^ate level of accoantabtllty. 

A given accotmtability tool, for example, a standardized test or an on-site 
evaluation, must be appropriately linked with the institution or person 
being held accountable. Aggregate or simimarized data are typically too 
gross to be used in Judging individual teachers, although they xaay be 
appropriatefy applied to institutions, such as districts or schools, or to all of 
a school's persormel. 
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Balancing Statewide comparabUitj with local ownership* 

Most States want data that let them compare schools and districts across 
the State. However, local districts and schools will use a State accountability 
system only If It produces information that also meets their needs and 
produces It on a time cycle that coincides with schooMevel planning. 
Balancing these two purposes Is both a technical and a political task. 



Expanding the altemmttres to tmdltlonal standaidlsed tests. 

The standardized tests that constitute the core of State accoimtabUlfy 
systems are Imperfect Instruments. Supplements and alternatives to these 
tests are In the research and development stage. States and districts should 
support efforts to devise altemattve Instruments and should compensate for 
some of the tests* limitations by using multiple indicators of student and 
school performance. 



Making fidr comparisons. 

States and districts are experimenting with ways to make comparisons that 
are both equitable and Informative. They must also ensure that 
comparisons are used In responsible and productive ways. In particular, the 
media should be acquainted with the control factors that States or districts 
are using so that they can communicate comparison results clearly to the 
public. 



Ensuring adequate cqMcity, 



Improving the 
Design and Use 
of Performance 
Accountability 
Systems 



states must ensure that adequate capacity— computer systems* stafT, 
training, and the like— Is In place before Implementing an expanded 
accountability system. 

Each State must devise an accoimtablllty system that meets Its own needs 
and political traditions. Therefore, the Study Group did not offer 
prescriptions for Improving these systems; rather, it presented three groups 
of general guidelines. 



Recommendations for designing better accountability systems: 



Make the purpose dear. 
Include multiple indicators. 

Select mdkrators that are derived from State or district goals; these 
goals should not be limited by today's tests and measurement 
technology. 

Include the indicators called for by the Council of Chief State School 
Offteers (CCSSO) among State indteators. 
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9 Make school- and district-level accountabllliy data public, and tailor 
performance reports to different audiences. 

• Ensure that local districts and schools can use the accountability data, 
beyond State rewards or punishments. 

• Establish an independent oversight body to monitor the development 
and Implementation of the performance accountability system. 

2. Recommendations for using accountability data to Improve low 
performing schools: 

• Use multiple criteria to identify low performing schools and districts. 

• Provide ppportimities for localfy directed Improvement activities before 
Initiating State intervention; if State intervention is considered 
appropriate, it should proceed in wdl-d^ed stages. 

3. Recommendations for using accountability data to sustain high 
performing or encourage slgnlOcantiy improving schools: 

• Develop 8 definition of what constitutes higli performance, and 
commtmlcate the definition to promote consensus among 
policymakers, educators, parents, and students in the State. Clarify 
the Importance of impr o v e m ent over time as well as ranking among the 
top schools in the State. 

• Use perfonnance measures that are objective and based on multiple 
Indicators, not on a sSnfSit test score, to identify high performing or 
significantly Improving schools. 

• If State recognition of performance requires an application process, 
provide assistance to help districts or schools apply. 

• Corislder a broad raiige of programs and iricentives. 

Accountability systems are powerful tools. In their current, rather primitive 
state, they produce some unintended effects that can constrain instruction 
and shape administrative policies in ways that mar^ find inappropriate. But 
they are rapidfy evolving as technology improves and as the political context 
shifts. Attention to their current shortcomings will ensure an evolution that 
wiU harness their power to help improve American schools. 
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Section 1 



Deeply held democratic traditions lie at the heart of public schooling in the 
United States. Although they are often diflOcult to put into practice, the 
notion of a common public Interest in schooling and the widespread belief 
that educators should be accountable to the larger body politic have 
endured, even as the nature of schooling and students has changed. The 
persistence of these values stems partly from the philosopl^ that aU 
government activity derives froai the consent of the governed and partly 
firom the practical belief that the more control citizens have over school 
policy, the more likely they are to support public educatloa 

Concern about accountability In education mirrors a growing interest In 
holding an types of public and private institutions accountable for their 
performance. The Federal Go/erxmient Is currently studying ways to define 
and report performance In a wide variety of public services. For exanq)le, 
the Federal Government now regularly reports mortality statistics for 
hospitals and the on-time departure records of the Nation's airlines. Two 
related assumptions tmdeilie the publication of this information: 1) that 
such data allow consumers to make more informed dedsftons and 2) that 
public reporting of performance informs sci ice providers and motivates 
them to improve their product. 

Accountability In public education has traditionally depended on such 
mechanisms for generatiqg information about schools as financial 
accounts, curricula, student attendance records, test results, and the like. 
It is assumed that such Information win serve as a resource for 
policymakers, concerned professionals, and the pubUc who can use it to 
demand or effect improvements in schooling. The fact that the 
overwhelming majority of States^ has chosen to structure accountability 
systems around the coUectlon and reportii^ of data is evidence of a 
broad-based consensus. 

Today's Accountability However, Uttle consensus exists about the kinds of specific data that 
Systems should be coUected or about how to report and use those data to hold 

schools accountable. The 50 States, therefore, exhibit considerable diversity 
in their accountability systems* 

• Some States assuine primary rrspoiisIblUty for coUectirigaiid report^ 
information to the pubUc, whfle others either share responsibility with 
kx^al districts or deh^ate authority entirely to them. 

• Some States assume that the unit of accountability Is the individual 
school: others focus on local districts. 

• Some States hold districts and schools accountable for student 
achievement In specific curriculum areas; others focus only on basic 
skins. 

• Some States assume that accountability extends Just tc the point of 
providing the pubUc with Information that can be used to bring about 
changes considered appropriate. Other States believe that 
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accountablltty consists not onty of rtportlng data about its schools, but 
also of linking that infonnation to specific policies that reward good 
performance or seek to remedy poor perfonnance. 

In addition, accountability systems are currently in a state of flux. More 
than one-half ttie States recently reported that they are considering changes 
in either their student assessment or overall accountability reporting 
programs. 

Ulis Report report is designed to assist State polk^makers who are interested in 
modifying or c ip a n di n g their educational accountability ^sterns. It resulted 
fitmi the deliberations of the Qfl9ce of Educational Research and 
Improvement (OERQ State AocountabOity Stucfy Group, whose members 
represented 10 States (CaUfomla, Cdorado, Connecticut Ftorida, Geoigla. 
Illinois, Massachusetts. Minnesota. New Jersey, and S^ vXh Carolina) and 
included governors* education aides. State l^glslatorB, legislative staff. State 
education agency staff, and representatives u local school districts? Hie 
report describes several different approaches to accountabiltiy. focusir^ on 
the ws^ that States and kx:al distrfcts use the data generated by different 
accountability systems, the consequences of that use. and the ways in 
which data use might be enhanced to further school improvement goals. 
The discussion is based on infbrmatkm obtained in a 1987 50-State surv^ 
conducted fay the Coundl of Chief State Sdiool OBUxm, the eiq>erience 
the 10 States re p rese nte d in the OERI Study Group, and research finding 
about the use and effects of State accountability ^stems in 4 of those 10 
States. 

One theme runs throughout the report: Hie design and use of an 
accountability system is hard and riddled with ambiguities. At one level, 
developing a system to collect data rdiabfy and tScteaOy poses a host of 
technk^al challeiiiges that are conq)llcated by the relatively primitive state of 
tests and measurement technok^. More iniportant. however, 
accountability involves tougli poUtfcal choices: Who should be held 
accountable— k)cal districts, individual schools, teachers, or students How 
much local diversity should be considered in the way perfonnance 
standards are defined? How much should States and local districts 
emphasize school improvement otijectlves relative to overset and 
monitoring? ff poUcy acttons are linked to the accountability ^tem. what is 
the proper mix of rewards, sanctions, and technk^al assistance? These are 
but a few of the issues that must be considered. Although this report 
provides guidance based on experience and research on mar^ of the issues, 
it carmot supply definitive answers. The Study Group recognized that that 
is no bestsjnstem. Each State must design an accountatdUty strata that 
supports its own educational goals and needs and is consistent with its own 
polltfcal traditions. 

The structure of the report reflects the purpose of offering general guidelines 
and presentix^ a variety of approaches that States might adapt to their own 
needs and drcumstances. Section 2 describes the basic elements of a State 
performance accountability system and outlines several conditions 
necessary for it to woilc effective^, regardless of its specific design or 
componoits. Section 3 describes aU generic types of accountability systems. 



how they operate in different States, their advantages and disadvantages, 
and the basic technical choices that States make in designing their own 
accountability systems. Section 4 shows the wdys in which Information 
generated by accountability systems is currently used by States and local 
districts and analyzes the consequences of its use. Section 5 summarizes 
major dOemmas that wHl continue to face policymakeis in developing or 
modifying State accountability systems. 

Following this overview and description of the anafytical context for the 
Study Gnnip*s recommendations, the Study Group's recommendations for 
how these systems might more effectively inform school Improvement 
strat^les are presented in section 6. 



AccoHlng to a 1967 Muvey conducted by the Council of ChScf Sute School Officers, all but Ave SUtcs 
(Alaalui. Iowa. Montana* Nebraska, and North Dakota) have provisions requiring that elthe: the State or 
local districts ooDect data about student perfbnnanoe. 

* The names and affiliations of the Study Group members are gh^ tn appendix A. 
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Section 2 



The Elements of a Performance Accountability System 

A number of mechanisms exist for promoting accountability in public 
education (table 1). These mechanisms serve different purposes and rely on 
different types of infoimatlon. Althou^ each pl£^ an Important role in 
holding institutiODS and individuals accountable, one mechanism, 
"performance repoitix^." has been selected as the focus of this study, 
because it helps the public learn how well schools are educating their 
students. Fiscal audits, school accreditation, and persormd evaluations 
have existed in public schools in one form or other for decades. Wtth the 
advent of Federal and State cat^orlcal programs, program evaluation has 
also become institutionalized as an accountability mechanlsnL Although 
standardized tests and other indicators of school performance have been 
routinely used over the past 20 years in assessir^ local districts, schools, 
and students, the notion of combining all these data Into a single system 
that reports overall perfonnarKx and then uses it as the basis for further 
policy action is relatively new. 

The examination of performance accountabULy systems b^ins with u 
description of the major components and the criteria that need to be met in 
constructing them. 

Indicators, or statistics that reveal something about the health or 
performance of the educational system, constitute the basic buildlr^ blocks 
of State performance accountability systems. However, not aU statistics 
about education can ftmction appropriately as indicators. Statistics qualify 
as indicators only if they serve as gauges, that is. if th^ tell a great deal 
about the entire system by reporting the condition of a few particularly 
significant features. For example, the number d students enrolled in 
schools is an important fact about the size of the educational system, but it 
tells little about how well the system is functioning. On the other hand, a 
statistic that reports the proportion of secondary students who have 
successfully completed advanced study in mathematics provides useful 
information about the level at which students are participating and 
achieving in that subject. This statistic provides considerable insight about 
the condition of the system and can be appropriately considered an 
indicator 

As they choose Indicators to include in accountability systems, 
policymakers determine who wHl be held accountable. /or u;hat and to 
whoni For example, if schooMevel scores on subject-matter tests are 
chosen as an indicator of student performance, policymakers are holding 
some entity (local schools, teachers, districts, or the State) accountable for 
teaching students a spedfk curriculum. On the other hand, if policymakers 
decide to use only tests of basic skills as the indicator of student 
performance, they are holding these individuals or agencies accountable on 
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Table 1. 

EDUCATIONAL ACCOUNTABILITY MBCHANI8MB 





Type of 


Whether 


PmpoM 


Use Information 


PnUlc^r Reported 


Fiscal Andlta 


Ensure that school 


Enforce fiscal probity Revenue and expendi- 


No. except in cases of 


funds are spent for 


ture data 


serious violations 



intended purposes, 
using sound accounting 
principles 



School Accreditation 



Ensure that all schools Enforce minimum School records, on-site No. except in cases of 

meet minimum standards visits serious problems 

resource and manage- 
ment standards; m^ 
include assessment of 
schocd oiiganlzation and 
peifomiance 



PeifKmance Reporttng 



Ascertain how well 
schools are providing 
quality educational 
services and producing 
desirable student 
outcomes 



Compare performance 
over time, against stan- 
dards, with other 
schools; reward, 
punish, and/or assist 
schools 



Statistical indicator 
data on resources, 
school oiganization, 
teachers, curriculum, 
and student outcomes 



Yes. in the txpectztkn 
that policymakers, 
educators, and the pubbc 
win act on the information 



PetMimel Bvahiatloa 



Evaluate how well 
individual principals 
and teachers are per- 
forming their Jobs 



Reward high perfor- 
mance; inform profes- 
sional development and 
tenure decisions 



Evaluations by super- 
visors; dassroom and 
school observations, 
woik portfolios 



No, because of the need to 
meet confidentiality and 
due process requirements 



Pra^mn Bvaluatloii 



Determine ni^ether a 
q>eciflc policy or 
program is producing 
its intended effects 



Decide whether to 
continue or modify 
program, determine 
future funding levels 



Record data, inteiviews 
with participants, obser- 
vations, various statisti- 
cal data 



Depends on public inter- 
est in the prpgram 



a very different standard. Slmllaily. if they choose to use only student 
perfonnance rather than to Include indicators of schooling resources and 
prc)cesses, the standard win be considendbly nan^ower than it wo^^ 
three domains were considered 

Criteria for The indicators States use In their accountability systems vaiy comlderabty. 
Desi£nin£ Indicator H^'^^^^^^* indicators included in ai^ system should meet six basic 
. criteria^ Thty should 



Systems 



1. Measure the central featurrs of schooling. 

2. Measure what Is ttctuBOy hckng taught or 
considered Important for students to know. 

3. Provide information that is pobcy-reicvant. 

4. Focus on the school site. 

5. AHow for £air coiiq>ai1sons. 

6. Maximize usefulness and mlnlmter burden. 

1 • Mewaie the eentnl Cntom of tchooHng. 

Performance accountability systems have traditionally focused on 
educational Inputs (for example, per pupil spendb^ and outputs (student 
test scored. That information can teU mdiether educational conditions are 
gettliig better or worse, but It provides little lnsi|i^t into why particular 
trends exist or how to fix problems or replicate successes. For example, 
recent test data Indicate that high school students have poor writing skills. 
But with only that information. State and local policymakers carmot answer 
some basic questkms: Do all students have this problem? Is It due to the 
nature and quality of the curriculum? Or is It because teachers lack the 
skills to teach writing effectively? 

Accountability is a bhmt tool unless policymakers, educators, and the 
pubUc have Informatkm that allows them to determine the likely sources of 
a problem and find dues about how to fix it. Consequently, accountability 
systems should do more than simply collect testing data. Th^ should 
provide an Integrated pkrture of the schoollr^ environment and Include data 
on fiscal and other resources, teachers, school organization, curriculum, 
arul the distribution of various student outcomes (achievement, 
participation) across different types of students and schools. 

2, MeMure iriiat is actiisl^ being taiaght or considered Important for 
students to know. 

Effective accountability does not require that States or local districts assess 
student perfonnance on all or even riiost of what is taught, but it does 
require that aocountablllty-related measures reflect at least a core 
component of what is taught Unless the standardized tests that measure 
student performance indude skills and content covered In the curriculum, 
schools win be held accoimtable for things th^ are not actually doing. 
Whether States choose to use baste skills or subject-matter tests, they need 
to consider whether what Is tested matches what is taught. 
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On the other hand, to the extent that widespread consensus exists about 
the skills or knowledge students must master, such as elementary science 
and writing. States should also make certain that schools are held 
accotmtable for such content, m^iether it is included in their own curricula 
or not. A broad-based consensus about what students ou^t to know is 
often difiknilt to reach, and the definitton of essential knowledge often varies 
consklerably from community to community. However, in most States and 
school distilcts. some measure of agreement exists about the core skills and 
knowledge that students of a given age should possess. Hie size of that core 
m^ vary, but the important point is that the perfomiance accountability 
syrs^tem should measure what the larger ccxmnunity believes is important fin- 
students to know. 



3. Prorlde Informatkm fhat Is poU^-relevant. 

Accoimtability systems should describe educational conditions that are of 
concern to poUcymakers and can be acted upon. For example, 
characteristk^s of teacher trainipg are policy-relevant, because these 
characteristics can be chapged through legislation or regulations governing 
teacher licenslpg. On the other hand, data about teacher attitudes and 
interaction with individual students, althou^ helpful in providiog a 
comprehensive picture ctf schooling, cannot so easily be acted upon by 
policymakers. 

4. Focus on the school site. 

Both research and practice have demonstrated that the individual school 
building is the fundamental imlt of improvement in public educatiorx. 
Therefore, accountability systems must collect data that can be broken 
down to that level. This crtterkm is partknilarly important fiar States with 
districts having schools that dlFer in cither their student exposition. 
oiganization« or leadership stjies. Accountability systems that blur those 
distinctions, for example, by using district-wide averages, make tt difiknilt to 
interpret data on educational performance or to use those data to inform 
school improvement strat^les. 

5. Allow for frlr comperisons. 

Given differences in student backgrounds and the resources available to 
schools, not all schools and students start out the same. Therefore, it is 
Important that any comparisons he made and reported fisdrly. The challenge 
is to reflect differences among schools and students fairly without 
institutionalizing lower expectations for some. 

6. M a rlmise Qsefnlness and mlnlmlge the Irarden. 

A critical measure of an accountability system's usefulness is whether it 
provides information that can be readily understood by a broad audience. 
Data must be oiiganized and reported in a way that is useful not only to 
educators, but also to policymakers, the press, and the general public. Data 
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must also be ttmefy and produced on a schedule that compatible with 
poliqrmakers* decision cycles. In addition, accountabilltyr system IndlcatoiB 
shoidd provide Infomiatlon about emeiglng problems such as the ^^hanging 
nimibers and types of students In urban areas. 

Minlmtziqg the burden requires designing a system that can be 
implemented within given cost constraints; does not strain current levels of 
State and local expertise in data collection, analysis, and use: and creates a 
limited respondent burden. Burdens can be reduced by coordinating State 
and local accountability efforts to minimize the amount of instructional time 
devoted to testiiig and other data collection, while ensuring that the 
information needs of both States and local districts are met. 

Hiese six criteria appty to the design of any State accountability system, no 
matter how it is structured or m^iat specific indicators are included. They 
must be met if the system is to produce accountability information that is 
reliable, fair, and usefid for improvir^ schools. 

One final element of an accountability system is inqilied in the word 
system No inatter how valid arid reliable ai^^sin^iiidicatorriiay be, it 
cannot be Interpreted independent of other indicators that reflect the larger 
educational systerrt An obvious aample is student achievement trends: 
Changes in student achievement over ttme carmot be interpreted unless 
there is information about other £actors, such as whether the 
characteristics of students takirig a particular test are the saine or whether 
the curricula, student attendance, or dropout rates have changed. 

Consequently, if an accountability system is to avoid reporting di^olnted 
pieces of information, it must be based on a model of how the educational 
system actually operates. Hiis model may be simple and Intuitive or it can 
be complex. Figures 1 and 2 show a siniple and a more complex model Hie 
simple system categorizes the major elements of the educational system, 
while the more complex model Illustrates relationships among those 
elements. These relationships do not constitute a model in either a strict 
predictive or causal sense, but th^ do serve as a framework, showing 
logical linkages anxnig dements of the schooling system, marqr of which are 
supported by research. 

Summaiy To be effective, an accotmtabillty system must allow users to view 
performance on any single indicator within the context of the larger 
educational system. Relying on a model (whether simple or complej^ that 
represents the entire schooling enterprise makes data Interpretation more 
valid and also increases the likelihood that accountability systems will 
inform school improvement efforts. 



Basing 
Accountability 
Indicators on a 
Schooling Model 



These criteria emerge both firom the experience of groups that uae educational Indicators 
and from reaearch on the design of sodal Indlcatora In a variety of areas, tnchidlng 
economic pobcy. education, health, aixi criminal Justice (for example, de Neufvllle. 1975; 
MacRae. 1965: Humane. 1967: Shavelaon et al.. 1987). 
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Figme 2.— A complex model of the edwettionel system showing links among elements 

SOURCE: Reprinted with pennteston firom pages vl and 32 Indbxxtor Systems fyrMonttortng 
Moffiemottcs and Science Educatbn (R-3570-NSn, August 1987, The RAND Corporation. 
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Section 3 



State Performance Accountability Systems: 
Description and Trends 

Types of Systems AU so states and the Dlstilct of Colunlblacone^ 

which can be used for accountabiltty purposes. Although all of these 
systems are Intended to provide information to help policymakers and 
educatoTB monitor the effects of policies and inqirove the quality of 
education* the systems vaiy considerably In their des^, largtfy reflecting 
the unique histoilcal and political circumstances of each State. Despite 
these differences, each syrtem can be classified aloqg two lines: 1) who is 
primarily responsible for the system and 2) whether the system is linked 
with other State poUdes. 

Systems differ in nt^ether the State or local districts have primary 
responsibility for deddiqg what data will be collected and how those data 
win be oiganized and reported. Most States have assumed primaiy 
responsibility, either as a result of spedflc legislation or in response to 
regulations devetoped by the State education qgency. TVvo States require 
that accountability data be collected and reported, but sOpviatt that k)cal 
districts must desdgn their own systems and report performance. In other 
States, the State and local district share accountability responsibilities. 

TWenty-five States with perfomiance accountability systems link those 
systems with direct State-level poUdes. In these States, performance data 
about schools and districts tri^er other State actions abned at improving 
education. Three general types of actions have emetged among States as 
they have formulated these links: 

1. Actions aimed at improving the quality of all schools. 

2. Actions meant to reward high performing or improving schools and also 
to provide incentives for others. 

3. Actions aimed at low performing schools. These actions are either 
sanctions that serve as disincentives to kw performance or efforts that 
assist in the school I mp ro v ement process. 

Many States publicly recognize high performing and improving schools, and 
a few use high performance to trigger more substantial actions. Common 
mechaisisms for recpgnltkm range finom publishing the names of high 
performing schools in newspapers, to honorir^ prbic4>als at luncheons or 
ceremonies, to presentlrig them with plaques, barmers, or other 
c ommen da t ions. Such activities provide public acknomdedgment, but th^ 
carmot be considered sustained pohcy actions. In contrast, seven States use 
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high performance to trigger more substantial actions or material rewards, 
such as allocation of extra resources or exemption from certain regulations 
or additional monltorlr^ requirements. 

In contrast to the relatively low number of States that link significant 
consequences to high performance. 25 States have developed policies that 
call for significant actions to be triggered by low performance or lack of 
Improvement These actions rar^e from forms of assistance to fairly severe 
types of sanctions. They Include requiring schools to develop plans for 
remedying their poor peiformance; providing technical assistance to develop 
and implement such pians; allocating additional funds to assist 
improvement efforts; wlthholdlr^ funds as a n^atlve consequence of poor 
performance; dedartaig 1)ankruptcy'' or "impairment** and intervening in the 
management of schools and dlsblcts; and consolidating poorly performing 
districts with others in the surrounding area. Some of these actions focus 
on schools, others on districts.^ 

Based on these two m^yor dimensions, focus cf control and policy Ibiks. six 
general types of peiformance accoimtablllty systems operating in the States 
can be identified. Their distribution is shown in table 2. 



Tride2. 

TYPES OF STATE PERFORMANCE ACCOTOTr ABILIIT 8T8TEM8 



l^rpe of System 


Totals 


Primary Responsibility 
State BOxed Local 


Policy Links 


25 


21 


3 1 


No PolKy Links 


21 


14 


6 1 


Totals 


46 


35 


9 2 



NOTE: Ftvc States do not report ooDectliig pcrformanoc data for acoountabllity purposes. 



Table 3 shows how these types of systems are represented among the 10 
States represented in the OERI Study Group. 



Table 3. 

TYPES OF 8 YBTEBflS IN OERI STUDY GROUP STATES 





Primary Responsibility 




Type of System 


State lOzed 


Local 


Policy Links 


CT. FL. GA IL. CA 


MN 




MA,NJ. SC 




No Policy Links 


— CO 
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The classification of States' accountability systems into types should not be 
interpreted to mean that the systems within a particular category are 
identical. States with the same general type can have quite HiftftimfTar 
systems. Moreover, many States are current^ e2q)anding or modifying their 
^sterns. These changes may actually alter their system 'Hype.'' For example, 
during the period of time the OERI Study Group met, the Colorado State 
Board of Education amended its accountability r^ulations, adding both a 
Statewide testing program and State reporting of local performance. These 
changes pushed Colorado fixxm a State that vests prlmaiy responsibility in 
local districts to a State in which both the State and local districts have 
major responsibilities for the system. Other States in the Study Group also 
reported impending changes. Nevertheless, the following snapshots of 
representative States provide some concrete examples of various types of 
systems, at least at one moment in time. 

State Responsibility. Nearty one-half of the States take primary responsibility for the design and 
Direct Policy Links implementation of their accoimtability systems (by action of either the 

legislature or the State board of education) and also link the performance of 
districts or schools ro other State policies. South Carolina has a highly 
structured system of this type, iii which uniform data are collected for an 
int^rated set of indicators on schools and districts throughout the State. 
Student-focused indicators include scores on norm-referenced tests of basic 
skills (that i&. Comprehensive Test of Basic Skills) and State-developed tests 
of student masteiy of grade-level objectives in reading, writing, and 
mathematics. Both types of tests are administered at several grade levels. 
School-focused indicatois include student and teacher attendance rates and 
student attrition (dropout) rates. 

In South Carolina, districts that fail to meet State-established criteria on a 
number of indicatora are labeled ''seriously impaired in educational quality" 
and are subject to State intervention. In the most extreme cases, they are 
subject to the withholding of funds. Schools that show 
greater-than-expected gains in student achievement receive monetary 
rewards. Moreover, the amount of the reward is increased for schools that 
also demonstrate gains in student and teacher attendance. Other State 
allocations, for example, funding for earfy childhood programs, are also 
based on the results of the assessment system (for example, the number of 
kindergartners in a district who fail to meet State criteria for first grade 
readiness). 

Geoigia has a less complex system that assesses students' basic 
competencies and achievement Statewide. It does not collect an integrated 
set of indicators about schools and districts, but State-level policy actions 
are nevertheless triggered by perfomiance on these assessments in that 
funds for remedial programs are allocated on the basis of students' test 
scores. This limited configuration is more common than the complex type of 
system found in South Carolina. 

New Jersey has one of the most highly publicized State-controlled systems, 
being among the first States to link performance data with policies 
permitting State inten^ntion in "educationally bankrupt" school districts. 
New Jersey monitors its schools on 51 indicators, including attendance. 
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teachers* certification status, extent of desegr^ation. and student 
performance on State tests. Policies link performance on these indicators to 
sanctions. Low levels of school performance set into motion an Incremental 
process whereby the State education agency closely monitors districts' 
efforts to improve their schools. At each level of the monitoring process, if 
Improvement does not occur. State action becomes Increasingly prescriptive, 
culminating In the State's steppiqg In and implementing Its own corrective 
action. In addition to linking sanctions to performance. New Jersey's 
policies also provide a tangible reward for high perfomiing schools: Districts 
with no deficient schools receive a 5-year waiver bom State monitoring, 
providir^g them considerable discretion in their prpgrams and operations. 

In contrast. Cormectkmt targets districts with low test scores for asslstarice. 
not sanctions. Current^, trends for remedial assistance, dropout 
prevention, and 'Apriority schools" (a technical assistance program) are 
linked to perfonnance on the Cormecticut Mastery Test. In 1989-90. 
Cormecticut's school finance formula will include test scores as well as a 
poverty index in a :)upll need component. This revision will direct more 
general aid to those districts with low performiiig pupils. 

State Responsibility, One-fourth of the States (14) take primai/ responsibility for the design and 
No Direct Policy links implementation of their accountability systems but have no other State 

policies linked direct^ to them. Nevada, for example, has recently b^un to 
collect district-level data about teacher/pupil ratios, ethnk: composition of 
the student body, and dix>pout rates, in addttion to Its achievement and 
competency testlr^ programs. However. Licst data are used only for a 
consolidated '^te of the State at Educatton** report to Nevada residents 
and do not directly trigger any other policy actions. 

Arizona has a less complex system, in which the legislature requires that all 
students be tested in basic reading and mathematics skills each year. 
Individual student scores are sent home with each student on the last day 
of the school year, and the State publishes summary statistics. No district- 
or school-level analyses are made, and no predetermined policy actions are 
triggered students* scores. 

Mixed Responsibility, Only three States share responsibility for the design and implementation of 
Direct Policy links the educational accountability system with k)cal districts and link their 

systems to other State policies. 

In Illinois, the responsibility for defining, collecting, and reporting IndlcatorB 
Is shared between the State and local levels. The State mandates that the 
schools develop local goals for ''excellence in education** and devise 
procedures for assessing student achievement at grades 3. 6. 8, and 1 1 . The 
State requires that, as part of these local assessments, some State designed 
assessment items be Included so that State-level results can be reported. 
The State Board of Education staff reviews and approves local plans and 
prepares school ^report cards" to provide assessment results to the pubUc. 
At the local level, districts formulate k)cal improvement goals and establish 
student learning objectives. Each district submits an assessment plan to 
the State. st4)ulatlr^ how it win determine whether students are meetlr^ 
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these goals and objectives at the specified grades. Local school districts are 
responsible for distributing the State prepared school report cards to the 
public. They also have the option of modifying these reports, as long as all 
the data and e}q)lanatoty texc are Included. 

Direct policy actions based on indicator results emanate from the State and 
local levels, lliese actions Include requiring schools or districts that do not 
meet their objectives to develop school improvement plans and providing 
technical assistance In the development and implementation of these plans. 

California also has a system with mixed responsibility and policy links. The 
State collects a standard set of quality indicators fixmi each district and 
reports uniformly for schools. It also mandates that each district develop a 
profideiH^ testfor high-achool graduatloa However, C^omia also 
encourages school districts to develop and report local indicators and 
suggests other data (for example, nimiber of writing assignments, amount 
of homewori^ that might be collected and reported. Whether or not these 
local indicators are actualfy developed remains the responsibility of local 
districts. State policies are triggered by both high and low performance on 
multiple State indicators. Hie normal 3-year review cyde is extended for 
high peifbrmlQg school9%.and technical assistance is provided to low 
performing schools. like many other States, however, California also 
publicly recognizes schools showing the greatest improvement on the State 
achievement tests and publishes lists of the poorest perforaiing schools. 

Mixed Responsibility, In six States, the State and local districts share responsibility for designing 
No Direct Policy Links and implementing the accotmtablltty system, but the system is not linked to 

other State-mandated poUdes. 

The Colorado State Board of Education requires that aU districts Implement 
a process for assessing and reporting on educational achievement at the 
local district kvd, giving locals almost total responsibility over the system. 
State rqgulatlons mandate that the process must specify goals and 
objectives, activities for achieving them« mechanisms for evaluating student 
perfomiance, and provisions for reporting results to residents of the- district 
and the State Board of Education. The State also requires that the plan be 
developed fay a group thattadudes at least one parent, one teacher, one 
school administrator, and one taxp^r, and it mandates that performance 
reports include infommtion about district priorities, academic achievement, 
dropout rates, and revenues and expenditures. Nonetheless, the 
substantive nature of the goals and activities and the vfsy in which 
evaluative information Is reported are locaDy determined and vaiy from 
district to district. 

As noted earlier, the Cotorado State Board has recently added more direct 
State responsibility to the accountability system with the development of a 
Statewide testing plan (based on experience with an aq)erimental Statewide 
testing program in 1986). However, while the new State tests are intended 
to be an integral part of the accotmtabllity system, it is not dear how they 
will link with the localfy deteraitned processes described above. 
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Even wJth this added layer of State responsibility, Colorado has chosen not 
to link peifonnance resialts to direct policy actions. Rather, the Intent of 
Colorado's accountability system Is to hold up districts with outstanding 
programs as examples to others, and to offer (but not mandate) technical 
assistance to schools and districts with serious problems. 

Oregon's system is another example of shared responsibility with no policy 
link Or^on mandates an achievement assessment designed and 
Implemented by the State, but It also calls for local districts to design and 
Implement high school proficiency tests. Althou^ the results of the 
Statewide assessment are used to identify schools needing improvement, no 
direct State policy actions are triggered by results from either the State or 
local tests. 

Local Responsibility, Only one state ghres local districts primary responsibility for designing and 
Direct Policy Links implementing their own accountability systems, which trigger specific State 

policies. Current State law in MIrmesota lets local districts select the learner 
outcomes that will be emphasized in instruction and assessed by 
perforaiance measures. However, Mimiesota requires that about one-third 
of the assessment items measure "core learner outcomes* that are specified 
by the State, and a State-developed item bank provides items for assessipg 
both core outcomes and those that are locally selected. Local districts must 
use the items measuring the core outcomes, but they also can elect to use 
others that match their local priorities. 

Despite its local focus, Mlimesota links si)ecific State policy actions to the 
results of the local assessments. The State requires that each district 
develop an individualized Assurance of Masteiy prpgram for ai^ student not 
performing to k)calfy set standards in mathematics and communication 
skills. However, the link between student performance data and State 
Invohrepient in the Assurance of Mastery program Is rather weak. Locals 
have a great deal of kitltude in the destgn and implementation of the 
Assurance of Mastery requirement, and the State has few mechanisms to 
enforce the policy. Although the State offers districts one dollar per student 
for compliance with the program, some districts find this an insufficient 
incentive. 

Only one State, Vermont, assigns local districts responsibility for designing 
and implementing their accountability systems, a competency testing 
program. Vermont has no State policies linked to the data. However, as in 
other States, the absence of State policy links does not preclude local action 
being triggered hv schools* performance on indicators!^ 

States that assign the prlmaty responsibility for the system to local districts 
or that have systems of shared responsibility are often States with strong 
traditions of local control, for example, Colorado, Illinois, Mlrmesota, and 
Vermont. The trend, however, even in these States, Is to move towaid 
greater State invohrement with accountability. Assigning the State primary 
responsibility often capitalizes on the greater technical capacity and 
resources at that level Additionally, State-controlled systems may be viewed 
as producing information that Is more useful to interested State 



Local Responsibility, 
No Direct Policy Links 



Weighing the 
Alternatives 



policymakers (for example, governors and State l^islators). because such 
systems can more easily produce comparable data about local schools and 
districts. A disadvantage of strong State control, however. Is that It moves 
the center of accountability away from local districts and schools— the units 
that are most able to use Information to Improve education. States that are 
developing shared responslbiltty are attempting to bridge this gap. As 
discussed in greater detail in section 5. States must weigh these trade-ofis 
careful^. 

States that choose to link policy actions to performance data do so in order 
to put teeth in their accountaUOity systems. PoUcymakers in these States 
generally believe that mandating specific consequences for various levels of 
performance provides incentives for improvement tangible mechanisms for 
helping schools inq)rove, or both. On the other hand, many States report 
that making performance data public is a powerful enough mechanism for 
spurring school improvement, and th^ have chosen not to link other 
specific actions with performance data. These States also make Individual 
ch oices based on their e3q)eneix:es and their assumptions about what is 
needed to improve their educational systems. 

Direct policy links have both advantages and disadvantages. Even poUcles 
aimed at rewarding high performance or improvement are not without 
detractors. Some consider ceremonies and awards to be superficial tokens 
that are inappropriate to the level of accomplishment schools have achieved. 
Others view them as cheap substitutes for more tangible rewards. Both 
recognition and more tangible rewards have been characterized as "dangling 
carrots" in front of educators; they sure seen by some as policies that 
embody the assumption that schools will rot work imless external rewards 
are forthcoming. Finally, in States that base rewards only on test scores, 
policies rev^arding high performance have been criticized as being oveiiy 
simplistic. 

Some States, however, report positive results fixnn recognition and rewards, 
and others are devising ways to keep these policies from being perceived 
simply as external nK)tlvation for hi^er testscoo^. Florida. Georgia, and 
South Carolina, for example, all note that a surge in school and district 
morale has accompanied recognition or rewards for schools that are doing 
well. Others have built into reward policies mechanisms to help keep them 
from being perceived as superficial or overly dependent on test scores. 
Massachusetts Project REACH (Rewarding Educational Achievement), for 
example, permits schools to apply for awards (currently $25,000 for 
elementary and $75,000 for secondary schools) and to base their 
applications on multiple criteria. Although the State requires that improved 
academic performance be one measure, it plans to include other indicators 
as well, some of which schools will select themselves as illustrative of their 
performance. 

The remedies or sanctions a State chooses are likely to reflect its past 
education TX)licies and prior efforts to impnjve low achieving schools. For 
example* Cormecticut has decided to provide extra resources to assist such 
schools, believing that the State's past history of fiscal inequities among 
local districts has contributed to current achievement differences. New 
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Jersey and South Carolina. In contrast levy stringent sanctions against low 
achieving districts and schools. Their policies stem from frustration with the 
limited gains that have been realized from past efforts to improve low 
achieving schools. In New Jers^. for example, the concern is greatest about 
the persistence of veiy low levels of perfomiance tn uifoan minority schools. 
In South Carolina, the greatest concern is with the continuing lack of 
improvement in small rural schools. 

The advantages and disadvantages of each type of policy for improving low 
performlpg schools must be considered carefully. Special resources targeted 
at low peiformerB may have the unintended consequence of providiiig an 
incentive to remain in the low achlevlpg category. Sanctions, particulaily 
the more severe ones, may affect the quality of the data that can be collected 
from schools. For examplt^ schools may fied pressed to do whatever is 
necessaiy to look good on perfomiance indicators, and this presstire can 
lead to such extreme actions as manipulation of the data reported to the 
State, and cheating by teachers and students during test preparation and 
testtakli^ 

Design Decisions The accountability systems described above flhistiate that as States 

Common to ^^^'^^^^^ aocountatriUty systems, they make a number of design decisions 
. Q . in addition to determinix^iKdio has prtmaiy responsibility for the fly^ 

Ail oy Stems and whether State actions wffl be linked to the peif onnance data the ^stem 
generates. These dedskms. listed in table 4. contribute %rther variety, even 
amot^ States with systems that are similar in terms of the locus of 
authority and policy links. 



Tdde4. 

DBCISKmB COMIION TO AIX TTFES OP FERFOIOCA^^ 
ACCOmiTABILnT STBIEMB 



• Whatwintheindkratorsbe? 

Student performance data? 

School resources and processes? 

Background data on students and communities? 

• Aa. what level win data be collected? 

State? Distrk:t? School? Classroom? Student? 

• At what level win data be aggr^ated and reported? 

Stat^ District? School? 

• What mechaiiisais win be used for reportiiig? 

Reports to the pubUc and media? 

Rqxnts to parents? 

Make data avaflable on request? 

o ^ schools be compared? 
Wtth an other schools? 
Wtth similar schools? 
Wtth their own past performance? 
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Data To Be Used Peiiiaps the most fimdaoieiital decisions States make are In 1) the choice 
as Indicators of indicators that will be used to J udgt and understand the perfozmance of 
students, schools, and districts and 2) the number of indicators that win be 
needed to provide the desired infonnatloa These decisions determine what 
Infomiatlon win be available to policymakers and educators. 

In section 2. six bask: crtterla for Indicators that are useful for school 
improvement were suggested. Few States have accountability systems 
whose indicators meet all these criteria. In many States, accountability 
rests laigely on data provided by tests of students* leanilQg outcomes. As 
yet,.the most common dedalonqypears to be that the core of the 
accoimtabiUty system is students* performance on tests of bask; academk? 
knowledge and skOls or minimum competencies. TWenty-nlne States have 
both achievement and mlninnnn competency testing requirements. 
Eighteen States rely on achievement assessments alone (either 
custom-made or commercial standardized test^. Only one State ctmently 
requires only minimimi competency data on Its students, and only three 
have no provision at all for Statewide student testing. In many States, these 
test scores are the only indicator data collected or reported. 

A 1987 survey by the Council of Chief State School QflOcers (CCSSO) found 
that only States go bqrond test scores and use an int^rated set of 
indicators.^ These States often Juxtapose data about students, school 
resources, and instructlcnal conditions to provide a more comprehensive 
pkrture of schooling. Callfbnila, for ezamide. uses a cooiblnatUm of student 
and school indkratora. Student mdfcators include scores on State-developed 
achievement tests, scores on Scfacdastfc Achievement Tests (SAT) and 
Advanced Placement lAP) tests, and first-year grades of students contirmlng 
in the State umverstty system. School Indteators Indude academte course 
enrollments, attendance, and dropout rates. 

Illinois reports a variety of student performance indicators, including 
percentage of students in each of the four quartlles on nationally normed 
tests, percentage of elementary students not promoted to next grade, h|gh 
school graduatton rates, Amertcan CoD^ Test (ACH scores, number and 
percentage of ACT test takers, and the natkmal percentile ranks schools' 
ACT composite scores, minois also reports a number of school-focused 
Indicatom (for exanq>le, dass size; hi^ school enrollments m math, 
science, EqgUsh, and social studies: race and gender of teachers; average 
years of teadilpg experience) and district-focused Indk^tors (for exan^le, 
average salaries for teachers and administrators and per pupil 
eaq)endlture4. 

Another 23 States repoxted that they collect performance data but not an 
integrated set of indkators. These States appear to base their systems 
nearly exclusively on student test data, although most of them also collect 
other types of data throt|g|i State accreditation, auditiiig, or teacher 
hcensing procedures. However, these other data collection processes 
typically operate independent^ of performance-based accountability 
SQTStems. T^le 5 shows the fi:equeni(y with whkrh the States in the CCSSO 
survey collect various types of data. 
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TddeS, 

TTFB8 OF DATASTAIES COLLECT 



Tjrpe of Dmta No. of States 



Student'focused data: 
Test Scores 

Achievement 45 

Competency 30 

Attendance 42 

Conqiletlon of School 44 

Postsecondaiy status 27 

School- and teacher-focused data 47 



(e.g.. eq;>endttures, teacher attendance, teacher credential status) 



Hie advantages of usti^ mult^le indicators are numerous. First, a single 
outcome measure provides onl^ limited infonnation on which to Judge 
perfonnance. For example, a State that uses only test scores may draw 
unwarranted conclusions about schools* effectiveness. Schools where large 
percentages of students become discouraged and drop out may produce 
high test scores, since the dropouts are often among those who score lowest 
on lests. Adding other outcome indicators, for exanq)le. the school's 
retention rate, and juxtaposing them with test scores provide a more 
complete picture oThow weD a school is doing. Other types of indicators, 
such as enrollments in advanced courses, can also provide va^ 
infonnation with which to evaluate the schocd, for example, the access 
students have to rigorous academic content and learning experiences. 
Second, if States collect outcome data only to assess performance, they 
provide policymakers with little understandirig about the schooling 
conditions imder which those outcomes were produced, and they offer little 
guidance about potential targets for improvement policies. 

While multiple indicators provide many advantages, there are also 
disadvantages in collecting too mar^ indicators. The expense and burden of 
data collection imposed on educators can become great, and the system 
itself can become unwiddy. States must therefore attempt to select a 
parsimonious set of indicators that provides enough information to be valid 
and useful for oversight and improvement. 

Inclusion of States also must decide whether to collect data about conditions outside 
Background Data the school, so that school performance indicators can be placed in their 

broader context. This usiialfy means collecting background information 
about students and their communities, for example, data about student 
mobility, socioeconomic status, and community wealth. Mar^ State 
poUcymakers believe that these data indicate the difBculty that schools and 
distrkrts face as they educate their particular student populations. For 
example, schools with high student mobility rates are expected to have 
greater dlfikulties carrying out long-term or sequential learning programs. 
It is not fair to hokl these schools entirely responsible for the learning 
progress of students who ruBy have attended many different schools over 
the course of a year. Other data, such as the flnaiicial resources available 
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localty, arc sometimes thought to be useful for undeirstandlng the 
constraints schocds face In the materials they can purchase or In the 
instructional services th^ can provide. 

Forty-five States In the CCSSO survey collect background data, but only 21 
repent that they use those data as contextual mfonnatlon to help 
policymakers and educators interpret performance indicators. South 
Carolina reports indicator data in the context of comparison bands 
reflectlr^ the percentage of scu<tents eUgfble for free limches at the school 
locally ^nerated flnandr^, and the percentage of first grade students 
scorlpg at or above the State standard on a Undexgarten readiness test. 
More typically, Illinois reports achievement indicator data Juxtaposed with 
data on student characteristics, for example, school enrollment by 
racial/ethnic cat^^xries, percentage of low-incocie students, percentage of 
limited-Eqgliah-proficient students, attendance rates, mobility rates, and 
percentages of students in academic, general and vocational tracks. The 
background characteristics reported in Illinois are shown in Qgure 3. 

Collecting and reporting relevant background data, however, is not easy. 
The proxies typically used to assess students* economic status are 
imprecise. Self-reported, student data are fiau^t with inaccuracies, as are 
principals* Judgments sdxmt family income and parental occupation* Data 
about the proportions of students eligible for firee lunch or who qualify for 
Federal compens^ / education assistance provide only gross estimates of 
the percentages of students who are poor and are unreliable as indicators of 
economfc status. Finally, when test data are set in the context of 
information about students* backgroimds. States must consider the 
possible unintended consequence of reinforcing pernicious stereotypes 
about disadvantaged racial and economic groups. 

Level of An important techrdcal dedston that States must make concerns the level 
Data Collection at which data win be collected This decision is critical because it influences 
the levels at which data can be organized and reported, and in some cases, 
it Influences the accuracy of tlie data. 

If, for example. States want schools to use performance data to assess the 
learning of tndbMual students, they must test eveiy student in the State, at 
least in certain grade levels, on all learning objectives of concern. But if 
States want to use student test data to assess the performance of 
classrooms, schools, dfstrtcts, or tfie State as a u)ho(e, they can use sampling 
strat^les. In California, for example, the school is considered the primary 
unit of accountability. Consequently, State tests are not expected to either 
diagnose or assess the learning of individual students, but rather to 
measure the performance of schools in helping their students learn. 
Therefore, California uses a matrix sampling plan, where each student 
answers only a subset of test items; together, students* scores reflect the 
performance of the school on all of the test items. Other States that want 
only school-, district-, or State-level scores simply collect data from 
representative samples of students. Of course, student level data can be 
organized to describe other levels. For example, Florida uses data about the 
universe of its students to create reports at the student classroom, school, 
district, r^on, and State levels. 
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Student Characteritttci 



Enrollment (Racial/Ethnic Background and Total) 





White 


Black 


Hispanic 


Asian 


Native 
American 


Total 
Number 


School 


98.7% 


.0% 


.6% 


.6% 


.0% 


154 


1 $UtKiiiL 












1 


district 


97.4% 


.0% 


1.3% 


1.3% 


.0% 


462 


1 St4it0 








i3t* 


.1% 





Low-Income Enrolment 

School 0 1-9% 
DIsWct flis^fe 
State 



29.1% 



Limited-Engiish Proficient Enrollment 

School n 2-9% 
District T 1-9^ 
State J 3.2% 



Student Attendance Rate 



School 
District 
State 



For tam e type 

School 
District 



95.6% 
96.2% 
93.8% 



94.9% 
95.5% 



Student Mob 

School 
District 
State 



llt3ti%) 



_^ 11.1% 

"6.8% 



21.2% 



% Of K8. Senior Clett who are: 
College Preparatory* General Education 
School |« School |« 



District | % 
State |« 



District I % 
State |« 



Vocational Education 

School |« 

District I % 
State « 



'The standard by which seniors are Judged to be prepared for college differs 
from high school to high school around the state. 



Flfare 3.— Backgioiiiid cluracterlatics collected by minola 

SOURCE: Illinois State Board of Education (1987). 
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An important rule of thumb governs sampling strat^ies: Data from samples 
canbe aggr^tcd up but n(^ down, lliat is. data sampled at the school 
level can usually be used to describe districts and States as well, but data 
sampled at the district level will not provide adequate information about 
individual schools and certainly not about individual students. 

These same considerations appty to data collection about classrooms, 
schools, and districts. States that collect data about school resources and 
processes from all schools can organize these data to provide information 
about an districts and about the State as a whole. Data, for example, course 
enrollments, that are collected about a unit from the unit itself, such as a 
school, mi^ also be more accurate, stcct those supplying the information 
may be in the best position to have the facts at hand 

The current trend is to collect test data about all students (this is done in 
40 States m the CCSSO report). Of the States that sanq>le students, some 
collect data that provide infomiatlon at the school level, some at the district 
level, and others at the State level States must consider the advantages and 
disadvantages of various data collection strategies. If student-levi^l 
indicators are collected from every student individual teachers and schools 
may find them useful for improving school programs and classroom 
instructkm if the measures are seen by locals as relevant to their own goals 
(a topic discussed later, in section S). However, the burden of such data 
collection can be great On the other hand, if school- or district-level data 
seem to be usefid for overstght and improvement then less burdensome 
samplir^ strategies can be used. 

Strategies for To a great extent the way States report data is closely linked to the wzy the 
Reporting Data data arc collected. Although States may not choose to release all possible 
combinations of performance data* they must decide whether and how to 
report IrKllcators. Typically, student test data have been reported to school 
administrators, teadiers. and parents, because these are the people who 
are intimately involved in the leamiqg process. However, with the increasing 
use of these data for accountability purposes, indicators (indudipg student 
test performance data) are increash^ seen as needed by the public and 
policymakers. Thus, decisions about the technical issues of collecting and 
oiganizir^ data for reporting are cHoatify related to the more politk^al issue of 
u;ho should be hekl acotniritfibte. 

Forty States, mdudiqg all of the States in the OERI Study Group, make the 
performance data th^ collect, at least at the State level, public. There is 
consensus that test scores of individual students should not be 
reported to the general pubUc. but many States provide these data to 
parents. However, there is considerable variety in the types of data that are 
reported to the pubttc and in how those data are reported. 

The national trend is to consider individual schools the primary unit of 
accountability. TWenty-fh^ States report schooHevel summartes of test data 
to the pubUc. Twelve States report district averages, but not schooHevel 
results. Only six States report State averages without district or 
schooHevd results. Corisiderable variety exists in u;hetha: how. and by 
u7hom indicators other than test scores are reported. Several States 




represented in the OERI study group report a number of indicators to the 
public In addition to test scores. California's arr^ of indicators for each 
school was noted eariien South Carolina reports student and teacher 
attendance and student attrition rates for each school. Connecticut reports 
percentage of twelfth grade students enterii^ college-bound courses, 
pupil/teacher ratios, teachers* salaries, per pupil expenditures, and a host 
of other fiscal data about school districts. Connecticut also reports 
indicators at the State level (figure 4). 

Among the States in which local districts determine the content of reports 
beyond test scores, Minnesota suggests that locals report local 
improvements at the district level while Geoigla requires districts to report 
the costs of providing schooUng localfy. 

A final way in which States difier is in /iou;pu2)l&: they make their 
accountability data Some States report school- and district-level data 
directly to the media throughout the State; others leave individual school 
data reporting to local districts, hi tMo latter group, some districts report 
school-level data to the media, and others send school reports to parents. 
Still other States keep a low profile, makipg their performance data publicly 
available only on request 

Each of these strategies for reporting data has consequences for schools. 
Most States' e3q>erience has been with the public reportir^g of standardized 
test scores. Many States have found the public to be highly interested in 
these scores, and not ahv^ for reasons relating to educational 
improvement. For exaii?ple, in many States, real estate agents have found 
published test scores useT'il in marketing homes in neighborhoods with 
high scoring schools. 

Generally, however. States report that h^ghty visible school-by-school score 
reporting has triggered considerable public and school district pressure for 
improving test scores. Some States also report extremely low morale in 
those schools where low perfomiance is made highly visible. In choosiiig 
reporting strategies. States must decide whether they desire these 
consequences and, if so, how they might be used as a positive impetus for 
Improvement Since schools will make considerable efiforts to look good on 
whatever indicators are made public, decisions about which indicators to 
report are as significant as how they will be reported. Whatever States 
decide about what indicators to report and how to report them, they must 
also anticipate how to avoid unintended and undesirable effects. 

Making Comparisons One of the most politically troublesome and technically sophisticated 

decisions States make is whether and how to compare the performance of 
schools and districts with one another. Many States do make and report 
comparisons, hi the CCSSO report, 23 States said that they actually report 
comparisons, and 16 more States said their reports enable comparisons to 
be made. 

Although the national trend is toward comparing the performance of 
schools, of districts, and of the States themselves, making fair comparisons 
is complex. Some States are able to compare their schools with national 
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Indicators 
of Success 



t 


Change is in the desired direction. 


\ 


Change is counter to the desired 
direction. 




"^here is little change. 




Baseline data. No comparable data were 
available before 1985-86. 




No data indicating change are available. 



Status 



Indicators of Change 



Commentary 



(9 



The number of handicapped 
students placed into com- 
petitive employment 

The proportion of vocational 
education program com- 
pleters who obtain a full-time 
job related to their training, 
pursue additional education, 
or enter the military 

The number of handicapped 
students graduating high 
school and placed into 
postsecondary training 

The number of handicapped 
students in vocational educa- 
tion programs 



Percentage of public high 
school students enrolled in 
the vocational-technical 
schools 

The average score on the 
general educational develop- 
ment (GED) tests 



In 1985-86, 1,805 of the 2,184 clients rehabilitated in pro- 
grams supported by the Division of Rehabilitation Services 
were placed into competitive employment. 

In 1985, 70.9 percent of all vocational education program 
completers either obtained a full-time job related to their 
training (24.3%); pursued additional education (43.5%); or 
entered the military (3.1%). This figure is up from 68.9 the 
previous year. 



No data are available on the number of handicapped stu- 
dents graduating high school who are placed in postsecon- 
dary training. Data will be available for the 1986-87 school 
year. 

19,457 handicapped students, constituting 12 percent of the 
total, were enrolled in vocational programs in the local 
public school districts in 1985-86. These data reestablish a 
baseline for this indicator, as the prior data were found to be 
significantly underreported. 

The percentage of public high school students enrolled in 
the vocational-technical schools in 1985 was 7.8. This was 
0.1 percentage points above the prior year and 2.2 per- 
centage points above the level of five years ago. 

In 1985, the average scores on the writing skills, social 
studies, reading, and mathematics sections of the GED test 
declined while the average increased slightly on the science 
section. The five components of the GED totaled 245.1 in 
1985 compared to 247.3 in 1984. 



Figure 4.— Connecticut's State-level Indicator reporting 

SOURCE: Connecticut State Department of Education (1986) 
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norms because they use commercially produced standardized tests. Other 
methods used for comparisons include ranking schools within the State and 
computing percentiles Into which schools or districts fall. More complicated 
approaches have been designed to make comparisons fair by creating an 
''even playing field* for all schools. These methods Include Judging 
perfonnance In terms of prpgress over past achievements, comparing 
schools and districts with similar characteristics, and predicting -expected" 
scores from a combination of past achievement and student demographic 
characteristics. 

Several different methods are represented In the OERI Study Group States. 
One di m e n sion of Florida's testlqg program Is a norm-referenced test. For 
this program, schools' percentile placements are computed and Individual 
schools arc compared to other schools that are similar in socioeconomic 
status and prior achievement. Hie average performance in the State Is then 
compared to national norms. 

California has devised a method of '1Ioatli^conq>arlsonbai^ a 
schod's test perfomiance is compared with that of 80 schools that rank 
Immediate^ above the school and with 80 that rank immediately bdow It 
on several background characterlstkrs (Qguie 5). 

CaUfomla also reports scores In the context of schools' past scores, as does 
Massachusetts. Massachusetts is amoqg a growing nuniber of States that Is 
developing r^resskm equations that predkrt expected test scores from both 
past perfonnance and school characterlstfcs. South Carolina compares 
schools In a number of different ways« Indudlpg past performance, the use 
of clusters of similar schools as comparison groups on ac^Alevement 
Indicators, and percentile rankings within the State on student and teacher 
attendance, dropout rates, and test scores (table 6). 



Tridee. 

BACKCnOUND CHARACTERISTICS OF SOUTH CAROLINA'S 
FIVE COMPARISON GROUPS: 1984-S5 



Gfoup 


Free Lunch 


Median Teacher Median Eqienae 
Median Percent Edncatlon Bqrond OverReqnlied 
At or Above Bachelor'a Degree EPAFnn41af 
Standard C8AB fjtm) ($/piq^ 


One 


76.2 


66.1 


0.9 


124 


TWO 


53.9 


66.9 


1.0 


144 


Three 


37.4 


74.5 


1.1 


170 


Four 


27.1 


75.4 


1.2 


179 


Five 


15.1 


78.5 


1.3 


246 



NOIE: Charactcilfttics were weighted aoooR^ 

nmtUpk le^etston for eM^ ^ade and the mibtests erf* reading ^ 

In predicted score* whidi were translated to ttandanUred values (Z-scores) and then averagod 

over grades for each year. In this way* schools were grouped on predicted, not actual 

performance. These snmge Z-vahiea were \ised to detomliie the gnoup pl^ 
schod. 

SOURCE: South Carolina Department of Education (1887). 
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Performance Report for California Schools 



C 12 NORCAL 
D 34!j67 midway UNIFIED 
S 8901234 EPIC SENIOR HIGH 
1986-87 Grade 12 Enrollment 



407 



Standard Score Display 



!3 



Course Enrollments 
1986-87 



1986-87 
School 

Star>dard 
Sco»» 



Comparison 
Group 
Standard Score 
Range 



School s Standard Score » 0 
Comparison Group Range « — 



200 



300 



_1_ 



500 



600 



700 



600 



Mathematics 

3 or more years 
Advanced mathematics 

English 

4 or more years 

Science 

3 or more years 
Chemistry 
Physics 

Advanced science 

History/Social Science 

4 or more years 

Foreign Language 

3 or more years 

Fine Arts 

1 year art/muslc/drama/dance 

Units required for graduation 

Unlverelty of Callfdmla Requirements 

Enrollments In a>f courses 
Graduates completing a-f courses 



488 
499 



652 



352-608 
359-^576 



368-615 



592 


349 


-624 


616 


337 


-596 


590 


351 


-614 


423 


347 


-615 


651 


344 


-636 


522 


352 


-5^3 


473 


345 


-617 


627 


369 


-627 


424 


336 


-592 


439 


35C 


-573 



Plfure 5.-— An cstmpk of CcUfornla't "floattng* iMuids 
SOURCE: Callfomla State Department of Edncatton (1987) 
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Table?. 

KKT DDSENSIONS OF THE SO STATE PERFORMANCE ACCOUNTABILITY STSTEBffiS 



S 

s 



S 0- 



lis 



38-2 
O ^ eg 



(0 



58 



§1 



0 

a 
o 
V 







Indicator Test 
System Type 


Public Report 


Compare Context 


Policy 
Links 


School 


District 


State 


Alabama 


state 


No 


Both 


Yes 


-I 

Yes 


Yes 


Yes 


No 


No 


Alaska 


None 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 




Of rk4>A 


No 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


No 


ATKansas 


State 


No 


Roth 


No 


Yes 


Yes 


ICS 


les 


Yes 




Mixeci 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Colorado 


Mixed 


Yes 


Achievement 


No 


Yes 


Yes 


Yes 


Yes 


No 


Connecticut 


State 


Yes 


Achievement 


No 


Yes 


Yes 


Yes 


Yes 


Yes 


Delaware 


state 


No 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


No 




State 


No 


Both 


Yes 


Yes 


Yes* 


les 


IMO 


Yes 


rionua 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Georgia 


State 


IMO 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Hawaii 


state* 


ICS 


Both 


Yes 


Yes 


Yes^ 


Yes 


Yes 


Yes 


Idaho 


State 


No 


Achievement 


No 


No 


No 


No 


No 


NO 


Illinois 


Mixed 


Yes 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


les 


Indiana 


State 


Yes 


Both 


Yes 


Yes 


Yes 




ICS 


ICS 


Iowa 


None 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


Kansas 


State 


Yes 


Achievement 


No 


Yes 


Yes 


No 


Yes 


No 


Kentucky 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Louisiana 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Maine 


State 


No 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


No 


Maryland 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


No 


No 


Massachusetts 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Michigan 


State 


No 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


Yes 


Minnesota 


Local 


No 


Achievement 


No 


Yes 


Yes 


Yes 


No 


Yes 


Mississippi 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Missouri 


State 


No 


Both 


No^ 


Yes 


Yes 


No 


No 


No 


Montana 


None 


No 


Achievement^ 


No 


No 


No 


N/A 


N/A 


N/A 


Nebraska 


None 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


Nevada 


State 


Yes 


Both 


No 


Yes 


Yes 


Yes 


No 


No 


New Hampshire 


State 


No 


Achievement 


No 


No 


No 


Yes 


No 


No 
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Indicator 


Test 


Public Report 






Pollqr 


state 


Level 


SjtUm 


Type 


School 


District 


State 


Compare 


Context 


Llnlrs 


New Jers^ 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


New Mexico 


Mixed 


Yes 


Both 


NO 


NO 


Yes 


Yes 


Yes 


Yes 


New York 


State 


Yes 


Both 


icS 


les 


xes 


Yes 


No 


Yes 


North Carolina 


State 


No 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


North Dakota 


None 


No 


Achievement^ 


No^ 


No^ 


Yes 


Yes 


No 


No 


Ohio 


Mixed 


Yes 


Both 


Yes 


YtS 


Yes 


No 


No 


No 


Oklahoma 


State 


No 


Achievement 


IMO 


Yes 


Yes 


Yes 


No 


Yes 


Oregon 


Mixed 


Yes 


Both 


IMO 


NO 


Yes 


No 


No 


No 


Pennsylvania 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Rhode Island 


State 


Yes 


Achievement 


Yes 


Yes 


Yes 


Yes 


No 


No 


South Carolina 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


South Dakota 


State 


No 


Achievement 


IMO 


NO 


NO 


No 


No 


No 


Tennessee 


State 


No 


Both 


IMO 


Yes 


Yes 


Yes 


No 


No 


Texas 


State 


Yes 


Both 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Utah 


Mixed 


Yes 


Both 


No 


No 


Yes 


Yes 


Yes 


No 


Vermont 


Local 


Yes 


Competency 


NO 


No 


N/A 


N/A 


No 


No 


Vbglnla 


State 


No 


Both 


No 


Yes 


Yes 


Yes 


No 


Yes 


Washington 


State 


Yes 


Achievement 


No 


Yes 


Yes 


Yes 


Yes 


Yes 


WestViiginia 


Mixed 


Yes 


Both 


No 


Yes 


Yes 


Yes 


No 


No 


Wisconsin 


Mixed 


No 


Both 


No 


No 


Yes 


No 


No 


No 


doming 


State 


No 


Achieveinent 


No 


No 


Yes 


No 


No 


No 


Totals 


S=35 


Yes=23 


A;=18 


Yes=25 


Yes=37 


Yes=43 


Yes=38 


Yes=21 


Yes=25 




L=2 


No=25 


C=l 


No=23 


No=ll 


No=4 


No=8 


No=26 


No=22 




M=9 


N/A=3 


Both=29 


N/A=3 


N/A;=3 


N/A=4 


N/A=5 


N/A;=4 


N/A^4 




None=5 




N/A=3 















BOVRCBl Council of Chief State School Officers 1987 Surv^ and related State documents. 

1 The District of Columbia and Hawaii each operate a single system In which the State and the district arc the same. 

2 Missouri and North Dakota send school— (plus North Dakota district — level data to parents but not to the press. 

3 Montana and North Dakota ollt / local districts the option of using a State achievement test 
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Each of these methods has its problems, as does the whole concept of 
maldng fair compailsons. Many oS the diflaculUes are the same as those that 
arise in reportiog peifomiance data in the context of student backgroimd 
data (see also section 5). 

Summary Each of the des^ decisions described in this section shapes the content 
and form of a State's accountability system. As the research suggests, some 
decisions lead to greater system effectiveness, make implementation more 
feasible, and generate more support. However, the decisions that lead to 
these desired ends differ amoqg States, depending on current political 
context and poUcy precedents. Consequently, no best set of design decisions 
can be prescribed for all States, and, not surprlsli^, while there axt 
trends, no two States have made identk^ decisions in all areas, l^ble 7 
Illustrates the dlverdty among the accountability systems of the 50 States 
and the District of Colimibia« 

It is too eaiiy to Judge how weD each of these systems will accomplish their 
di^al goals of oversight and educational iiiq)rovement However, some 
insight can be gained from anaifyzlng the experience of four of the Stady 
Group States. Those data are reported in the following section. 



1 Another daM of policy acdm taii^ tndMdi^ 
other ccltate 0br caGBOiple. ^)Mles. coune* oompfeM 

receives (acttdemlc or vocattonii) orwbcUieraatiidentreoelveaadtploiiiaormcidr«oertll^^ 
aUendanoe. 

2 De^itte Its stioi« State feciM. South Cmllna to 
proce>a.WthtaaState- in a Twto*fJ fan^^ 

proceaa and co n te nt of the tmpruveiucnt plana they develop. Additlanal^. local *f1 t^ K' la are given 
reaponaibill^ for leportfa^ achool-kwd pertananoe data to their oomin unities. 

SMlnneaota'aaccountriJltygratantocuntnflybetog i g^ ^ 

diai^ magr modl^ tta local fcoia. Ihe auflr^ cfa^^ 
'coreleanicrqutaMMrttcniaandpMblksattaiofflicaeco Mpgi ^^ 
Implemeiited* IfbmaoCa'a localbr orten 
State and locala ahare o^or reapomirfhty. 

4 Althoi^ not a perfDrmanoe reportSi« q^atem. M«a^ 
•Imllar. loca% oriented acoountabfUty fkam cw oik . 

5 Ihe CCSSO auivqr provided nadooal data on acGoimtabffity data ooilectkn a^ 
andthc DtotrW of Cohnnb^ 

reported In uito paper. 
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Section 4 



■ 



Uses and Consequences of Perfonnance 
Accountability Systems 



What Difference 
Do Accountability 
Systems Make? 



The dlflferent ways States can organize their accountability systems and the 
basic choices they need to make in designing them have been surveyed. 
This section examines how local districts and schools in States with 
accoiintahilrty systems actually use the data th^ produce and how the 
systems influence school and classroom activities. 

In designing perfomiance accountability systems. State policymakers make 
certain assumptions about how the resulting data win be used for State 
purposes and also by local districts and schools. For exan:q>le, one 
cQnqxment of Florida's testlqg program requires that the individual student 
must perform at certain crtabUshcd levels or a high school d4)loma, and 
possfbty grade level promotloru wHl be denied. Remedial instruction Is to be 
provided miienever students are not mastering required skills. The 
programs also require schools, districts, and the State Itself to demonstxate 
educational achlevemeiit espedaUy over tlriie, or fsu» 
enibaiTassinent through release of the data. Finally, the Florida programs 
provide sufficient Informatkm for educators to continuously review the 
curriculum content and instructional programs and acQust their programs 
to Improve educational services. 

However, the way the accountability systems are used and the effects thqr 
produce depend on the response of local educators, namely, how th^ 
choose to Incorporate their requirements and the information generated by 
them into school practice. This process significant^ influences the extent to 
which accountability systems actually operate as their designers expected 
and the extent to which they produce unintended consequences. 

To answer the question of how accountability systems are mfluencir^g 
schools and classrooms, extensive interviews were conducted with more 
than 350 poUcymakers axui practitioners in California, Florkla, Georgia, and 
Mlimesota, at the State, district, school, and classroom levela^ Th^ were 
asked 



• How State accountability systems affect school-level activities and 
classroom teaching. 

• What kind of resource burden (time and mon^) the accountability 
systems imposed. 

• How accountability results are used. 

The investigation shows that accountability systems produce mixed results. 
They are vexy powerful policy tools that have significantly changed school- 
level plaimlng and teaching activities. But in that power lie some potential 
problems. Many reqxmdents felt that the current systems motivate 
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State-Based 
Perfonnance 
Accountability 
Systems 

Instructional Effects 

I iMj rerj dote attention to the BState test] lesolts. Teet 
resuUs are discoMed wtth the CKoltj cloeelf. and the 
raratta are known throoghont the achooL I tell teachers 
what la In the teatt, and he^i them derdop materlala that 
win help benefit us. We want Uda to score well feel proud, 
and be proud. I want the comniunlty to think we are doing 
a good Job. 

and 

I review the reanttn of theae teata very carefiill7. 1 look to 
aee If children are aGhierfaBg aa well aa we expected .... I 
aaaeia the need for remediation in apedflc aWUf by doing 
an item analjala. Theae data are used by junior high 
atndenta and aides wlio tutor our students on individual 
Itema . . . . Wte hare aidea who work directltjr with children 
to remediate deficient Items. Ihese aides are directed a 
teacher to work with deficient children lAo score low on 
the tests. We have ftntr of theae aides in outbuilding. 

and 

Those scores baalcally define our direction in the schooL 
We anatyae the acorea and a^fnst our teaching 
according]^— for ezamptet if our students score low on the 
use of apoatrophes. I Just finished writing my goala for the 
coming school year, by which ru be evaluated. Part of 
those goals are based on the school's performance this 



teachers and adminlstjnators to focus narrowly on the specific content of 
tests and create incentives to manipulate their results. However, precisely 
because the systems have such power to change behavior, they hold the 
potential for dra m a tic all y enhancing the school lnq)rovcnient process. As 
accountability ^tems evo^e, especially if they incorporate more 
sophisticated measures of student achievement and stress other indlcatora 
as wen as achievement \hxy can inform the overall schooling process. 

The investigation also shows t at the experience of the three States with 
strong State components diflfers from that of the State with a locally 
designed and operated system (Minnesota). Thus. State and local systems 
are treated separately in this discussion. 

Pdfomiance accountability systems have caused local districts and schools 
in California. Florida, and Geoigia to cha^ige the w^ they plan and teach. 
However, these chaqges typically are not dhected at lnq)rovement. in the 
broader sense of using data to pursue institutional and currlcular goals 
more effectively. Rather, accountability systems have caused principals and 
teachers to change thcta- behavior because they want their students to 
perform weU on State tests. 

The comments of several principals are Illustrative: 
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year on the Basic Skills mod Fimctloiisl literacy tests. I 
know that we are weak In writing composition, so one of 
my goals for the coming year Is to Improve students* 
composition skills. 

Like principals, a majortty of the teachers In the flddwork sample are 
concerned about their students' test peifonnance and are willing to change 
their teachiqg practices to improve it. The following sentiment, expressed by 
a middle school teacher, is typical: 

There is more thrust to teach toward the test. We also 
make an effort to teach test-taking skills. I don*t feel that 
the material on the test Is so Important to teach, bat I 
feel It Is necessary to teach that material so the students 
win do wen on the test* The highest stress time Is during 
testing week. I want my Uds to achieve and be snccessfeL 

Many of the teachers interviewed said that they spend time in drill and 
practice aimed at improvix^ test scores (for example, daily instruction on 
specific skills that ait covered in the test): teaching test-takiog skills, such 
as how to complete an answer sheet and work imder time con8tiair.ts: and 
aligning the curriculum with the tests. Teachers express substantia J 
ambiralence about "teaching to the test," that is, teaching skills or content 
that is covered on a State test In the way it appears on the test 

On the other hand, they have little or no problem with teaching test-taking 
skills. Florida districts have made particularly extensive efforts to assist 
students who perform pooily on State tests. Students who are in danger of 
failing the functional literacy test receive special tutoring and are often 
placed in smaller remedial classes. In addition to providing special services 
for these students, schools implement schoohvlde activities to inq>rove test 
scores. For exanq)le, one person said: 

When we were deficient two years ago, we identified 
deficiencies bj Item. Wt spend 30 mfamtes a week on 
skins. Wt came nine points two years In a row. We 
worked on fire areas, Inchiding fractions and roondlng 
numbers. On those we did well, and that brought ns up. 
It*s a total school effort. This school has a wide unj of 
activities to Improve scores, snch as mlnl-lessons once a 
week and videotaped wh<de school lessons to Improve 
scores. Bat are we concerned with the school reputation 
rather than the Uds? AU tai an, though, the State 
requirements make ns concentrate on things and try to 
Improve. 

The extent to which an accountability system afifects school practice difiers 
across schools. The lowest achieving schools tend to change their practices 
more than higher achieving ones; and elementary schools react more than 
high schools, largely because of the basic skills focus of State tests. 
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Resource Burden Tiie time and OKmey spent m taking tests. coUectlng the data, and 

reporting the results vary across districts, but generally the resources 
required to administer tests do not impose a aignfflr^nt burden. In most 
schools, the system Is effectively 'bigaiized so that teachere are bothered 
little by test logistics. The real resource Issue is the amount of Instructional 
time schools report spending to prepare students to take the test. Most 
districts have their own testing systems which run parallel to the State 
programs, thus considerably Increasing the time students spend taking 
standardized tests. 

With regard to reporting, school personnel perceive the need for making fair 
comparisons across different kinds of schools. However, attempts to provide 
fialr comparisons are not ahvays successfid. For example, although 

California uses a 'conqKulson band' (figure 6) to make test results 
compar able across schools, not everyonfr-elther In the schools or In the 

communlty^-has accepted the notion that a similar ranking has the saine 
meaning across bands. 

Results Data fixra»C^ornla.Ftorlda, and Georgia suggest that state accountability 
systems are producing some effects consistent with State poUcymakers' 
intentions. Local districts and schools take the existence of such systems 
seriously and have changed their behavior In order to Improve perfomiance. 
However, that performance has been narrowly defined by the focus and 
scope of State tests. 

Although school personnel are actively responding to these systems, mariy 
do not view them as uc-fid or valid measures of their schools' perfomiance. 
Fewer than 20 percent Ust test scores as the most Important source of 
informatton about a district or school; Indeed, half the superintendents and 
principals In the fiddwork sample did not even mention State test scores as 
among the major sources of Information they use to determine how wdl 
their districts or schools are (kring. Their response to these systems stems 
firan the considerable pressure they fed about having their students do 
well. Tills Is the case even where there are no explicit rewards or 
punishments attached to test scores. Just the posslblltty of looking bad" {for 
example, appearing on a "deficient' list or receiving adverse publicity in the 
media). 

Local superintendents and principals recognize the need to use other 
Indicators as well as test scores in Judging a district's or school's 
performance. They typkally use a mix of quantitative and qualitative 
indk^ators such as graduation rates, the number of students attending 
collie, teacher attendance, dassroom observations, talking with students, 
and overall school atmosphere. However, they perceive the State as being 
primarily concerned with test scores, and consequently they have phfltmHrd 
much of thdr energy into Impiovlng performance on that one Indkator. This 
response has occurred even in the two States that report having an 
integrated set of indkators. whteh suggests that if States want locals to pay 
attention to other indicators (for example, course enrollments, teacher 




quality, the distribution of educational outcomes across different kinds of 
students), they will need to emphasize these indicators more strongly, 
pcriiaps by stressing them when reporting comparisons. 



Localty Based 
Accountability 
Systems In 
Operation 



Hie accotmtablllty systems in California, Florida, and Geoigla appear to 
have minimized the Iqgistlcai burden of data collection and testing for local 
schools, but the amount of instructional time used for State and local 
testing programs and for review in preparation remains an issue. 

Minnesota has a locally based accountability system. The results of 
interviews suggest that in districts in Minnesota where school ofiSdals are 
committed to using accountability data to inform a school improvement 
process, the system influences school practice more broadly than It does in 
States with a State-operated system. However, \)rtiether this process occurs 
or does not depends entirely on local priorities and capacity. Same districts 
use the data cztenslvety to infomi prsbctice, while others do little more than 
meet State requirements to test 

Some districts collect a varied of data in addition to test scores-4br 
cxanq[>le, surv^ of student and parent attitudes and foUowups on recent 
graduates. The district piiblic^ reports this infonnation and helps 
individual schools use the data to guide their school improvement plans. 
Other districts meet State reportiqg requirements, but th^ tend not to use 
the data once it has been published 

Local districts also vary considerabfy in their responses to the State 
Assurance of Masteiy requirements. One district in the fieldwork sanq>le 
has established an elaborate accountability process that includes a 
districtwide l^gh school con:q>etency test to determine whether or not 
students are to be granted diplomas, and a set of student outcome 
otjectlves that teachers are required to incorporate into their teachiqg. This 
latter measure requires teachers to si :bmit weekly lesson plans and to be 
prepared for random classroom observations by teams of administrate 
who will make certain that the teaching is linked to the district's student 
outcome objectives. On the other hand, another district in the sample has 
not yet implemented any program to meet the Assurance of Masteiy 
requiremeiits, and yet aziother has designed a strategy that teachera, 
counsetors, and principals believe will have little effect on the way low 
achieving students are taug^it. 

In practice* the effects of some locally designed accountability systems on 
school and dassroom practice may be similar to the effects of 
State-operated systems. For example, teachera and principals in the 
Miimesota district that has an elaborate accountability process may see 
thelrbehayior asdicumscribedby the focus and scope of tests, as do their 
coUeagues in Florida, Georgia, and California. Tliedifferei^ the 
accountability standards and procedures in Miimesota have been set tiy the 
local district rather than tiy State policymakera. On the other hand, in those 
Miimesota districts that, tsy preference or capacity, collect and use only 
limited data, the accoimtabiUty system has veiy little effect on principals* 
and teachers* behavior. 



1 llita dtocuMlon is baaed (m inteivlcw daUl collected ta fou^ 

conducted by the Ccntor for PbttcyReseanOi In £ducaUoi)(CI^ 1985 and 1987 on the 

Implcmentatlan and eferta of a variety of educational refenn poUdea. focusiqg on State eflbrta to monitor 
educational pro^neaa. Field tntovlew daU were collected by reaeaichcn at RAND. Rutgen Untvenlty. and 
the Unhnenlty of Wlaconatn-lfadlaon. Intcnrlewa were conducted in U local diatrlcta and 40 achoola In 
theaefi9urSUtea.Aiipnntiiiatcly 120 SUte-levdreapondenU were interviewed in the aprti^t of 1986: they 
included govemon' aides. SUte leglalaton and their atafls. SUte Board of Education members. State 
Department of Education offldab. and interest ^up repreaenUdves. In each SUte. staff rcsporafcle for 
the State acoountablhty pn^^am wa« interviewed. Interviews in local districts a^ 
in Pcbriary-March and Ma3^0une 1987. RespondenU included: supeilntendenU. achool board memben. 
district .)ei«onnel adminlstratoiB. the district teatlr>g director, principals, counselor*, and 83nie 120 
classroom teachers. 
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Section 5 



Unresolved Dilemmas 

Tlie task of designing an effective accountability system and using its 
results appropriately Is hard and riddled with aiiiblguitles. System 
development Invoh^ a series of technical decisions (for example, which 
Indicators to Include and the level at which to collect data). These decisions 
have far-ranging political Implications. For example, how much support the 
system will ei^qy among various constituencies, the funding available to 
support It. and the balance between State and local concerns must be taken 
Into account In Its operations, an accoimtablllty system may generate 
consequences different fixmi those Intended by Its designers. 

Many of these technical and politfcal Issues can never be completely 
resolved, but careful attention to the major problems polkymakers face In 
developing or modifying such a system will Increase Its overall effectiveness. 
This section examines six unresolved dilemmas. The choices they pose 
Involve core political values such as the amount of control higher levels of 
government should exert over school and classroom activities and the 
appropriate balance between "carrot* and 'stick* approaches to 
accountability. Since consensus on these Issues Is rarely easy to achieve, 
the problems discussed here constitute some of the toughest challenges 
that system advocates and designers of accoimtabllity systems face. 

Inherent In the concept of accountability Is the notion of 
oversight— monitoring to ensure that public monies are spent in ways that 
produce acceptable levels of educational performance. But Judging overall 
quality can play only a limited role in helping schools If it cannot also 
remecfy identified problems and enhance organizational effectiveness. Most 
State accountability systems, at least implicitly, have oversight and 
improvement as their dual purposes, but the improvement function has 
often been a poor second. Although States and local districts have espoused 
the notion that accountability systems should both assess performance and 
inform the improvement of practice, thi^ have made little use of 
accotmtability data beyond efforts to Increase test scores. Recent initiatives 
that seek to strengthen curriculum frameworks and textbooks and link 
standardized tests to them, and that use accountability data to allocate 
resources and improve instruction, are b^lrming to equalize the two 
functions. 

The challenge is to structure an accoimtabllity system that provides 
Information that is useful for improvement azid still makes dear that 
oversight is taken seriously. This can be achieved through 1) the types of 
data that are coUected. reported, and used and 2) the mix of rewards, 
sanctions, and technical assistance linked to the accountability ^^em. 



Balancing Oversight 
and Improvement 
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Overst^t and Improvement win be more evenly balanced If infonnatlon In 
addition to test scores is emphasized (for exaixq)le, the distribution of 
course-takirig across different types of students or students* postsecondary 
at t ai nment ). As discussed in section 4, States send strong sfgrjalswitli their 
decisions about what data to report and haw to use those data 
Achievement data wOl ahvs^ constitute the core of any accotmtabllity 
system. However, making other information available and Identifying it as 
valuable greatly increases the probability that accountability systems can 
Inform school inqmivement in the broadest sense. 

One way to supplement achievement data is to integrate eidstir^ data more 
effectively. States typically use such mechanisms as accreditation, fiscal 
audits, and teacher licensure commissions to collect a variety of data 
b^ond what is available from their accountability systems. However, they 
rarefy integrate these additional data irito a slr^ system that could i^ 
the entire schooUng process. Ihis need for integration also extends to units 
within State departinents of education and kxral district offices. For 
example, there is often little coordination arrxnig testipg and evaluation 
bureaus, offices reqxmsfble far tedmk:?! assistance to categorical programs 
or for general school inqmivement anc staff development units. 

Maintaining an appropriate balance between oversight and improvement 
also requires polity decisions about the proper mix amor^g rewards, 
sanctions, and technical assistance strategies. Since States have onfy 
recentfy linked significant policy actions to their accountability systems, it is 
too early even to suggest what the optimal mix might be for differ^ types of 
States. There is vlrtualfy no Information on the effects of using 
accountability system results to reward h<gh achievir^ schools or to provide 
special resources for low achieving schools. It is not known whether awards 
are primarlfy symbolic or Aether they purchase capacity and motivation 
that are not praslbk without theia Neither is tt kriown what the optima 
award level for sustaining performance without causing serious 
inequities across districts arid scluxds Is. reseandi and eq^^ 
dearfy demonstrate the need for focused technical assistance, yet the 
relative effectiveness of different forms of assistance, when tied to 
accountability data. Is still largefyunknowa Similarfy, States have little 
experience wtth nxxre severe forms of sanctions, such as intervention in 
k)cal district management. It is known, however, that sanctions are inherent 
in accountability systems, but no one wants to kx)k bad, and everyone will 
work hard not to do so, even withoul txphat sanctkms. 

Given these uncertainties. States are likefy to experience a period of 
e3q)erlmentatlon where they wHl try new approaches, whk:h they win then 
need to modify in light of subsequent research and experience. 

Determining the 
impropriate Level of 
Accountability 



Althou^ State govenmients have constitutional authority to provide public 
education* they share responsibility for implementir^ this authority with 
local districts, indivklual schools, principals, teachers, and even students 
themselves. The division of reqxmsibility among these institutions and 
people is sometimes quite clear. For exarrq)le, responsibility for adequate 
flnandr^g rests with States and local districts because they are the onfy 
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Institutions with taxing powers. In other areas, the division Is not as 
apparent. For example. States that now grant high school diplomas on the 
basis of competency test results are assuming that students are ultimately 
responsible for meeting certain standards. However, in requiring districts 
and schools to provide remedial assistance, they are also expecting them to 
assume some responsibility for student performance. The tough question is 
how much responsibility each player should bear, particular^ when 
negative accountability results can profoundly shape students* lives. 

The question of who should be held responsible for what outcomes becomes 
a msyor issue in matching an accountability instrument (for example, a 
standardized achievement test or on-site persotmel evaluation) witti the 
Institution or person being held accountable. The problem has recently 
crystallized around the issue of linkir^ teacher and student performance 
and whether one can use indicator data such as scores on standardized 
tests to determine appropriate conqiensation for teachers. 

Matching teacher and student performance is very difiBcuIt. Research shows 
that variability in educational practice and student outcomes is often 
greater within the same school than across mult4)Ie schools. Therefore, 
aggregate data or data that can be summarized statistically are typically too 
gross to be used in evaluating individual teachers. Student test scores are 
not valid proxies for teacher performance, given aU the intervening factors, 
for example, whether the test measures what teachers teach, whether 
teachers are assigned sutjects for which th^ are trained, what knowledge 
and skills students brir^ with them to the classroom, and whether a 
particular student is even in the teadier^s classroom for the fitll year. The 
data collected must be tailored to what individual teachers actually do ffor 
example, how they teach), which usually requires a variety of different kinds 
of information, including classroom observations. 

On the other hand, indicator data are appropriate for holdii^ accountable a 
State government, a local district an individual school, or the principal and 
teachers representirig a school as a group. As pressure for greater 
accountability in public education mounts, policymakers and the public 
must understand the purpose and limits of different accountability 
instruments. Those Instruments can serve as fair and reliable measures of 
performance only if they are appropriately matched with the unit belr^g held 
accountable. 



Balancing 
Statewide 
Comparability with 
Local Ownership 



The issue of balancing Statewide comparability with local ownership goes 
straight to the heart of mazimizir^ the usefulness and minimiadng the 
burden of a State accountability ^stem. For a variety of reasons, most 
States prefer some comparability of data. The people are interested not Just 
in the schools in their own local communities, but in the State as a whole. 
Taxes support aU those schools, and concerns about the link between 
education and economic development extend beyond the local community. 
Furthermore, comparable data permit State policymakers to Judge the 
similarity of educational opportunities and quality across different districts 
and types of students. This comparison is particular^ important in States 
with heterogeneous or highly mobile populations. 
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At the same time, local districts and schools will use a State accountability 
sjrstem for their own purposes only if the type of information needed 
Statewide Is also consistent with local needs and priorities. Several decades 
of research on the implementation of "top-down" policies has shown the 
luportance of local ownership in influencing whether or not a policy 
produces its intended effiects. For example, if State accountability data are 
to be used to inform local improvement efforts, the system must not only 
contain the type of infomiation teachers need to assist individual students, 
but it must also produce those datr on a time cycle that coincides with 
school-level plannLig requirements (for exanq)le, in assigning students and 
allocating resources). 

At one level balancing these two purposes is a technical challenge, that is. 
to obtain comparable data that arc suflQciently detailed cost-eflBdently. At 
another level, however, it is a political task, that Is, to reach a consensus 
between State and local policymakers and between pollcymakere and 
practitioners about the purpose and priorities of an accountability system 
that can serve diverse constituencies. This consensus does not yet exist in 
most States As indicated earlier, the need for State-level accoimtabllity and 
comparab?^ lata dominates systems in which the State has primary 
responsibility, and even in those with locally operated systems, pressure is 
growing for greater comparability across districta 

One concrete action that States can take in the short term is to coorxllnate 
State and local testing and other data collection activities. California is now 
conducting a pilot study of a comprehensive testing program that would 
combine the State system, which produces only school-level data, with k)cal 
testing systems designed to obtain data on all students. Another strategy for 
coordlnattag State and kwal data collection is Mirmesota's use of a test item 
bank that includes items measuring core learner outcomes that all districts 
must use, as well as thousands of other validated items that local districts 
and individual teachers can elect to use for measuring local curriculum 
results. 



Expsindtng the 
Altemativrs to 
Traditional 
Standardized Tests 



states might also consider designing accountability systems that allow some 
measures to be defined kwally or others to be included or not at local 
dlscretioa If these varied k)cal data were collected within the ftamework of a 
common model of schooling, they could be linked to data that were 
comparable across the State, and their relationships reported in a 
consistent, if not alw^ uniform, marmer. 

Large-scale testing programs form the core of State accountability systems 
and, often drive the local response. Yet, they are at best imperfect 
instruments for measuring academic performance. Questions about the 
current status and use of standardized tests are being raised increasingly 
by researchers, the media, and citizen groups (for example. Congressional 
Budget Office, 1987; Savage, 1984; Friends for Education, 1987). The most 
common criticisms include the imiblllty of existing tests * o measure the full 
range of achievement, particularly higher order skills: the limited overlap 
between the substance of standardized tests and the skills and content 
stressed in widely used textbooks; a general narrowing of the curriculum as 
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a result of the tests* emphasis on basic skills; and the confusing messages 
transmitted to policymakers and the public as a result of the long Intervals 
between revising nomis of standardized tests. 

One way to redress the limits of standardized tests has already be . 
suggested: Accoimtablltty systems should rety on multiple Indicators of 
student and school p^^rformance. Dropout rates, the number and type of 
students enrolled In different courses, and student attitudes and 
aspirations are aU examples of irdicators that can be used to supplement 
standardized test scores to measure the health of the education system. 

A number of States are also experimenting with enhancements of traditional 
standardized tests. For example, California, Connecticut, and Minnesota 
districts (figure 6) are now using writing samples to test student writing 
skills. In assessing students* abilities to apply rules of language usage and 
to present logical arguments, these tests can provide more valid data about 
thinking and reasoning skills than can a multiple-choice test. However, the 
use of writing samples on a widespread basis would probably require States 
to spend more to ensure reliable scoring. 

Other supplements or alternatives to standardized testing are now in the 
research and development st^ge. The Educational Testing Service (ETS), at 
the request of the National Science Foundation (NSF), is designing a test of 
science achievement which presents students with a set of laboratory 
apparatus and asks them to perform a scientific task. Althou^ such an 
approach m£^ measure aspects of scientific achievement not tapped 
previous^, it is difficult to administer and score, and it is also costfy. 
Consequently, this technology is not yet cost-effldent for large-scale data 
collectiorL Other alternatives include using computers to simulate tasks 
that require the application of higher order skills, and using teacher 
Judgments, collected through a structured survey instrument, to 
supplement standardized test scores (for example, see Calfee, 1988). 

States and local school districts are not typically in a position to provide 
total funding for the research and development efforts necessary to develop 
alternatives to standardized tests, but they can motivate others to share 
that responsibility. Just as States have pressured textbook publishers to 
Improve the quality of their products, they can demand that corrmiercial test 
publishers develop tests that measure higher order skills more validly, and 
that they revise the norms of their tests more fipequentfy. In addition. States 
and local districts can encourage Federal agencies responsible for research 
and development, such as the Department of Education and the NSF, to 
devote more resources to designing enhancements and supplements to 
standardized tests. They can also make certain that the research and 
development efforts of these agencies are sensitive to the constraints of time, 
cost, and expertise within which States and local districts must operate 
their accoimtabllity systems. 

States and local districts are struggling with the problem of how to compare 
districts and schools fairfy. No single approach has emerged as the optimal 
strategy. Most people agree that it is ority fair to take into consideration 
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Wrtttm Language Evaluation 
SMior High Compoaita Sample # 1 
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Writtan Language Evaluation 
Senior High Cc ipoaite Sample # 2 

T 

Y Y 

Y 




WPTSINBOCOVSMEPSP 

Learner Outcome 

VHP « Writing Process (Pre-writlng. writlno, revision) 

IS m Thesis statement or main idea 

IN m Introduction 

BO - Body 

CO m Concfusion 

VS m Variety of sentence patterns 

ME « Endmarks. internal punctuation, and capitaliza- 
tion (Mectianics) 

P « Pronoun case and number 

SP - Spelling 



Learner Outcome 



WP 
TS1 
TS2 
TS3 
VS1 
VS2 
VS3 
ME1 

ME2 



Writing Process (Pre-wrlting, writlfig, revision) 
Topic sentence, paragraph 1 
Topic sentence, paragr^ 2 
Topic sentence, paragraph 3 
Variety of sentence patterns, paragraph 1 
Variety of sentence patterns, paragraph 2 
Variety of sentence pMems. paragraph 3 
Endmarks. internal punctualton. and capitali* 
zatkKi (Mechank:s). paragraph 1 
Endmarks. internal punctuation, and capitali- 



zation (IMechank^s). paragraph 2 
ME3 - Endmarks. internal punctuatkMi. and capita 
zatkm (Mechanics), paragraph 3 
PI - Pronoun case and numtier. paragnph 1 
P2 > Pronoun case and number, paragraph 2 
P3 > Pronoun case and number, paragraph 3 
SP - Speliing (0% to 20% errors) 



IMKEY: Y-Yes N-No 



Written Language Evaluation 
Senior High Composite Sample # 3 
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Learner Outcome 

SI « Ust four sources of hifbnnatton on assigned 
topic 

BE m Gather suffk:jem informabon to make bibifo- 

graphicai entry 
00- Include direct quotatfon from one of the sources 

BMKEY: Y-Yes N«No 



TESTINQ PROCEDURE 

in April 1967 all 12th grade students were 
given one of three writing assessnnent tasks to 
perform. Six of seven seniors completed 
Samples 1 or 2. and one out of seven seniors 
participated in Sample 3, a research exercise 
in the media center. 

The readers evaluated 100 student writings 
(50-Tech, 50— Apollo) of Sample 1 and 100 
student writings (50— Tech, 50 Apollo) of Sam- 
pie 2. A total of 68 Sample 3 student writings 
(35— Tech, 33 Apollo) were also evaluated. Stu- 
dent writings were selected randomly but in 
the proper ratio from the three levels of 
courses offered to students. Using specific 
criteria, the readers tabulated the results of 
the writing samples. 

Sample 1 was designed to assess seniors' 
abilities to use the writing process to com- 
pose an essay with thesis (main idea) state- 
ment, introduction, body, and conclusion. 
Readers also evaluated students' abilities to 
generate a variety of sentence patterns and 
use proper writing mechanics. 

Sample 2 was designed to assess the 
abilities of seniors to utilize the writing pro- 
cess to compose three paragraphs: narrative, 
descrlpth^, and parsuashw. Readers also 
evaluated students' abilities to generate a 
variety of sentence patterns and use proper 
writing mechanics. 



Sample 3 was designed to assess how suc- 
cessfully seniors could locate sources of In- 
formation about an assigned topic In the 
media center. The task also included gather* 
Ing the Information necessary to make a direct 
quotation from a source and make a 
blbik)graphy entry. 

COMPOSITE OF RESULTS 

The composite charts on page 3 and at right 
are n oiimmary of the writing sample results. 
The learner outcome number and parallel test 
questtons folk>wed by percentage results are 
listed In the first three columna. 

Test results cf 80% accuracy met the 
minimum standard; those within the range of 
80«% accuracy vm and exceeded expecta- 
tkms; and those over 90% exceeded expecta- 
tkKis. Test results below 80% accuracy dkf 
not meet expectations. 



Ffgufe 6.— An altematlTe to stindatdlsed, nmlttiite-cholce tests 

SOURCE: District 742 Community Schools. St. Cloud. MN (1987) 
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significant differences In resource levels and in the students being taught at 
different districts and schools. However, tr&nslatlng that principle into 
practice is diflBcuIt For txaanplt. the use of predicted scores can 
insUtutionaliase low e3q>ectations for some schools; comparison bands may 
make some schools fed they are "the best of the worst." Some States and 
districts also report that because the people tend to think in absolutes, it is 
confusing to grasp the controls for wealth and student characteristics, 
especial^ when schools viewed by the community as low performing are 
designated wortl^ of merit. Some States and districts have e3q>erlenced 
additional problems when they have given extra funds to schools that enroll 
more diflOcult to educate students, because, in some cases, the additional 
funding has proven a disincentive to improvement. On the other hand, 
failing to control for differences in such fiactors as student mobility or the 
number of non-En^h-speaking students seems to violate basic notions of 
fairness and a eates disincentives for serviqg those in greatest need. 

States continue to seek more valid approaches for comparing districts and 
schools. For example. South Carolina is using multiple methods to deal 
vi^th the dilemmas that comparisons present. To relate their performance to 
thai of other schools, schools in South Carolina can compare themselves 
with a group of schools haviqg similar characteristics (table 5). However, in 
making school incentive awards, the State ccaaxpaxes schools with their own 
performance over time, using expected perfomiance levels based on a 
matched longitudinal anafysis of the test scores of students in that school. 
California has also sought to inq)rove its method for comparing schools. Its 
original approach was to group schools in five comparison baiids based on 
student characteristics, but this put schools at the top or bottom of a band 
at a disadvantage. By using fbatbig comparison bands, the State gives each 
school the opportunity to conq)arc its performance with that of 80 more 
advantaged and 80 less advaitfaged schools. 

States and districts continue to exp erim ent with improved technical 
methods for comparisons. However, the comparisons must be used in 
responsible and productive ways, especially in the media*s reporting of 
comparisons. In some States where controls for school context are used by 
the State education department in its reporting, the laigest newspapers in 
the State con Jnue to report oniy raw test scores; in other States, however, 
the newspapers regularly place schools in context when reporting their test 
scores. If a State or district decides to build control fiactors into its 
comparisons, the news media must be acquainted with those factors and 
with how to clearly commimicate their meaning to the public. 

The expansion and heightened visibility of accountability systems requljne 
new roles for State education agencies, local districts, and individual 
schools. However, many State goveniments pay little attention to the 
resources and capacity these new functions require of each oiganization. 
For example, a State operated accountability ^tem designed to inform 
school improvement activities assumes that State departments of education 
can move beyond their traditional role of testing students and simply 
"cotmting things* (numbers and types of students, program offerings, 
libraiy size) to assessing the entire schooling process and assisting local 
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districts in the use of their data. Yet this new charge comes at a time when 
mary state agencies are . iplementlng a variety of other reforms. Some 
have faced budget and staff reductions because of decreased Federal and 
State funding, and many are constrained by the staffing requirements of 
cat^orlcal programs. 

Similarly, accoimtabiUty systems place new responsibilities on local districts 
and schools. Many large urban districts and affluent suburban ones have 
traditionally had the capacity to collect and use data in ways designed to aid 
school improvement However, In most States, local capacity levels vary 
significantly. These differences are particularly marked in States with small 
rural districts or especially poor ones. States need to consider whether 
adequate capacity— whether it be computer systems, additional staff, or 
trainir^g— is in place when initiating an expanded accoimtabiUty system. 

Not onty are these six dilenmias difficult to resohre. policymakers will need 
to return to them repeated^ as the firontiers of technical knowledge about 
indicators improves and political preferences shift States, districts, and 
individual schools will have to sedc consensus on basic political questions 
civAiO is accountable for which outcomes, and they will also have to find 
new ways to use information about schooling effectively. 

Because the use of performance data fi-om accountability systems has been 
neglected tradlUonally b^- States and local districts, the OERI Study Group 
focused most of its recommendations on that issue. These r^ommendations 
are summarized in the next section. 
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Section 6 



The Study Group's Recommendations for Improving The 
Design and Use of Performance Accountability Systems 



Designing Better 
Accountability 
Systems 



The OERI Study Group holds the firm belief that State accountability 
systems camiot realize their full potential as policy Instruments luiiess they 
can be used to Improve schools. Reporting to the public is a critical 
mechanism for ensuring that schools are answerable for their performance. 
But if the use of accountability data stops with newspaper articles or 
reports to parents, attempts to remedy poor performance or sustain high 
adiievement may be haphazard, ineffective, and short-lived Accountability 
data should be collected and used in ways that reflect and inform the entire 
schooling process— how schools are organized, what is taught, how it is 
taught, and what Is learned. 

The Study Group also recognizes that each State must devise a system that 
meets its own needs and political traditions. Consequently, the 
recommendations offered here provide general guidelines for States to 
consider in strengthening the link between data use and improved 
schooling. The guidelines accommodate both the diversity and complexity of 
State accountability systems. 

The Study Group's recommendations fall into three categories: 

1. General suggestions for designing accountability systems that serve all 
students; 

2. Ways to incorporate accountability systems into a strategy for improving 
low achieving schools; and 

3. Ways to sustain high achieving or significantly improving schools. 

Appendix B presents a list of people whom States can call upon in 
addressing many of the challenges associated with designing and using a 
performance accountability system. 

Blake the purpose of accountability syitema clear. States vary 
considerably in what they expect their accountability systems to 
accomplish, but whatever those purposes are. they must be communicated 
clearly. One source of variation is the decision about who is held 
responsible. Some States have decided that local districts and schools 
should be held formally accotmtable for students* mastering a set of basic 
skills but have left: the scope and content of curricula entirely to local 
discretion as long as at least those basic skills are covered. In an effort to 
make curricula more uniform across districts, policymakers in other States 
have decided that the State should establish a set of frameworks specifying 
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a broad lapge of cuirlcula content and then reflect those specifications In 
its tests. Other differences in purpose stem ftiom whether or not States 
decide to link policy actions to their testing prc)gran]s and what those 
actions are. However, whatever the State's purposes may be. they must be 
communicated to principals and teachers. Teachers* concerns about 
teaching to the test" could be greatly assuaged if teachers knew exactly 
what the State expects testing programs to accomplish. 

The purpose of a State accountability system will be much clearer If 
poliqm[iakers. educators, and the public carefully consider the uses they 
want such a i^stem to serve before they begin modifyirig or ncp^n^fng the 
one currently In place. It Is necessary to decide what Is to be done with data 
orx:e they are collected (for example, use them for purely Informational 
purposes, link them to State or local polky. use them to Inform school 
practice). It Is also necessary to consider \iidiat type of system Is most 
compatible with sodal and political conditions in a particular State. Finally, 
oriice consensus Is reached about the system's purpose, sufficient 
development time should be allowed to design the accountabOtty system 
and its component indteators to be consistent with Its Intended use. 

Include multiple Indicaton. Whether an accountability system is used to 
reward high performance, remedy poor performance, or inform an overall 
school improvement process. It should collect data on mult4)le indicatorB to 
provide an integrated picture of the schooling environment. Student-related 
indicators rright Include: 

• Student course-taklrig and. for elementary students, the amount of 
time spent on different subjects; 

• Attendance: 

• Promotion and dropout rates; 

• Proportion of vocatlorial education studeritsrneetlrig academic 
requirements: 

• Proportion of studeritsrrieetlr^uriiversity eritrarice requiremerits; 
Student participation in the arts arid extracurricular activities; 

• Student attitudes and social behavior (for «mmplf>, vandaHgm); i^nd 

• Student achievement broadly defined to include writing skills, higher 
order skills, and subject matter competence. 

These data should be be broken down by sex. ethnicity, and socioeconomic 
status so that performance can be monitored for different groups of 
students. Despite difficulties in collecting data on students* socioeconomic 
status, the Study Gmp recommends that States and local districts seek 
valid proxies and collect such data. 

To provide a comprehensive picture of the schooling context. States shouM 
also consider collecting data on 

• Teacher quality: 

• Fiscal resources and their allocation; 

• Administrative leadership; 

• Curricula; 

• Local assessment* evaluation, and plarming systems; and 

• Commtmity support. 
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Select indicatort that arc aUgned with State or district goals and are 
not limited by today's tests and measures. To strengthen the connection 
between accountablltty data and school improvement activities, 
accountablllly systems should be linked to State and local educational goals 
and to current programs for achieving those goals. Citizens and parents, as 
well as educators, should be Involved In establishing performance and 
Improvement goals. 

Policymakers also need to base the design of an accountability system on an 
Int^rated model of schooUi^. Such a model can provide clear guidelines for 
adding new indicators and replacing others as measurement technology 
Improves {for example, with the development of better tests of hl^er order 
skills or measures of teacher quality) or as State and district goals shift. 
Accountability systems need to be dynamic in responding to new policy 
priorities and in appfylqg new technologies, but States must make certain 
that in changing individual indicators, their systems still measure the 
central features of schooling in consistent and meaningful ways. 

Include those indicators called for by the Council of Chief State School 
Offlceis If poaaflUe. The CCSSO Indk^ators (listed in appendix C) include 
measures of student outcomes, school context, and educational policies and 
practices. The CCSSO plans to report data on these indicators for all States 
by 1990, as part of its continuing effort to design h4gh*quallty educational 
measures that Fxt conq)arable across States. 

Make school- and district-Ierel accoontablllty data pnbUc and tailor 
pettoiiu aace reports io dtflSerent audleneea. PubUc disclosure lies at the 
heart of democratic accountability, whether the news is good or bad. But 
disclosure means more than Just publishing data; it also requires making 
performance reports understand^le to the various publics concerned about 
schools— policymakers, educators, parents, the media, and the general 
public. 

Effective public disclosure requires devoting substantial time and energy to 
pladng raw data in context and helping various audiences to inteipret 
them. AccoimtabUtty systems do not produce unequivocal Information 
about the condition of schooling, and most of the data wHl be susceptible to 
various interpretations. By thou^tfuDy analyzing and reportlr^the fuU 
range of responsible interpretations. States and local districts can help their 
various audiences avoid jumping to hasty or simplistic Gq>lanations and 
conclusions and can encourage real school improvement 

Ensure that local districts and scho<ris find some use for accoimtabillty 
data, beyond State rewards or pmadahmenta. States, districts, and 
individual schools have very different information needs. Information 
sufficient to determine whether State or local performance accountability 
standards are being met is often considerably less detailed than the data 
schools and teachers need to improve programs or tMagnno^ the 
Instmctlonal needs of individual students. However, to the extent that State 
accountability requirements are consistent and coordinated with local data 
needs, the system will be less burdensome and more useful to principals 
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Using 

Accountability Data 
To Improve 
Low Performing 
Schools 



and teachers. Local districts and schools need to feel that the accountability 
^rstem is not an unwelcome or irrelevant intrusion from the State, but is 
useful for informing their own decisions and activities. 

Establish an independent oveitlght body to monitor the derelopment 
and Implementatton of the perfimnance accountability tjrstem. This 
body should include policymakers, educators, educational testing and policy 
e?q)erts. and representatives of the public. Its functions should be to 
communicate the purposes and rationale for the system clearly, monitor Its 
technical quality, including student outcome measures; to assess Its effect 
on school practice and curricula; and to recommend needed chaqges to the 
legislature or State board of education. Such a body is particularly 
important in view of the hktfy changes In testing and measurement 
technology that will occur over the next decade and the shifts in priorities as 
accountability systems become a more central part of the educational policy 
landscape. 

Use multiple criteria to Identify low performing schools and districts. 

Performance should be Judged both in absolute terms, such as meetipg a 
defined standard, and relatively, by gaugiqg change over time (for example, 
the extent to which a school or district has improved relative to its past 
performance). Particular attention should be paid to the distribution of 
perfonnance across different schools within a district and different types of 
students within a school. Once districts or schools are designated as low 
performing, additional data that can inform the improvement process 
should be gathered bom on-site reviews. However, the actual designation 
should be based on statistical indicator data, because such data are viewed 
as comparable and reliable across districts and thus less subject to political 
pressure. 

Provide sufficient opportunities for locally directed improvement 
activities before Inltlatiag State intervention; if State intervention is 
comidered appropriate, it should proceed in well-defined stages. Some 
States find any type of State actlvtty beyond technical assistance or financial 
aid Inappropriate, given their political traditions. Others, however, consider 
State-mandated Improvement activities and even intervention to be proper. 
No matter what role is considered appropriate, locals shouM have ample 
opportunity to improve their own peiformance. and the State's improvement 
activities should move through progressive steps: 

1. Inform local districts and schools of problems that are Identified, so that 
educators and boards of education can take action; 

2. Inform citizens so that they can take action; 

3. Target additional resources and assistance; 

4. Mandate Improvement activities (for example, require districts to develop 
a remediation plan and then monitor their progress on It or withhold State 
funds or require increased local fiscal effort); and 
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5. B^ln State intetvention (for example, direct involvement in district 
management, including the removal of local administrators). 



Using 

Accountability Data 
To Sustain 
High Performing 
or Encourage 
Significantly 
Improving Schools 



Some States m^ find the first two or three activities desirable; others may 
find some of the activities in the fourth cat^oiy suited to their needs and 
strat^es; still others may find direct intervention appropriate as a last 
step. However, no matter how £ar a State moves on this continuimi. it 
should make certain that it moves sequentially and gives local districts 
adequate opportunity to respond at each step. 

E«ch State must develop its own deflinition of what conttltatet high 
perfimnanee, but in developing such a definition, it should make certain 
that a common understanding exists among all relevant parties 
(policymakers, educators, parents, and students). Policy action taigeted at 
high performii:^ schools is intended to maintain and encourage that 
peifomiance. However, such achievement may be defined in a variety of 
w^: against an absolute standard, in comparison with similar schools, or 
in terms of significant improvement over time. States may wish to base an 
incentive program on any or all of these notions of high performance, but 
whichever they choose, the selection criteria must be well understood This 
is particulaiiy true if the State decides to reward significantly lnq)roving 
schools. Members of the Study Group caution that the public needs a dear 
understanding of selection criteria that designate a school with tow absolute 
test scores as "high performing* when scores improve significantly. 
Therefore, it is critical that notions of inq)rovement and gain be clearly 
stated if th^ are used as criteria. 

Any identification of high performing or lignlflcantl^ improving 
schools thoidd be ol^ectlTe and bwed on multiple Indicators* not on a 

single test score. Indicators of special relevance for high performing schools 
might include: 

• Measures of higher order skills and writing ability; 

• Course enrollments, including the proportion enrolled in advanced 
placement classes; 

• The proportion of students meeting university entrance requirements; 

• The proportion of vocational students meeting academic standards; and 

• The distribution of performance among varying student groups. 

As with all indicators of school performance, the State shoukl take into 
consideration how different types of students are achieving on each of these 
indicators. The Study Group also recommends that States may want to 
consider using additional verification finom other soim^s such as site visits. 

If State recognition of performance rests upon tn application process, 
the State (or another Mgeacyi should offer technical asrittance to 

districts or schools that require help in applying. Such assistance may be 
particularly important in States with locally defined goals and might be 
provided by intermediate service units, universities, or other oiganizations 
close to the local level. 
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Consider a broad range of prognuns and Incenthres. Among the strat^es 
that might be used to sustain high perfomiance are cash awards, less 
frequent State or local monitoring, release from some State r^ulatlons. and 
resources to use in assisting other schools. All of these approaches require 
careful consideratloa For example, cash awards can be problematic if they 
are too high (creating serious inequities among schools) or too low 
(providing an InsufBdent incentive). The State also has to decide how award 
money can be spent In some States. Incentive grants can be used only to 
supplement the school budget; in others, funds can be allocated as salary 
bonuses to school personnel Similar^, in^ile the provision of resources to 
high performing schools to assist less successful ones m^ create useful 
models. States also need to ensure that this process reflects knowledge 
about successful strat^les for change and irmovation. 
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Afterword 



Making 
Performance 
Accountability 
Systems 
Responsible and 
Responsive 



The e3q)erlence of schools and districts across the Nation dearly 
demonstrates that accountability systems are powerful tools. Teachers, 
administrators, and local communities respond to tlie data such systems 
produce, particularly to the Implicit message embedded In the nature of 
current outcome measures. like any other powerful tool, these systems are 
as good or as bad as their design. In their current, rather primtttve state, 
they appear to produce some unintended effects that may constrain 
instruction and shape administrative policies in ways that many find 
Inappropriate. 

But performance accountability systems are dynamic. Hicy will 
change— and must change— as indicator technology evolves and as the 
political context shifts. Intelligent efforts to improve accountability systems 
will help to ensure that th^ will evolve In ws^ that harness the systems 
appropriately to Improvement strategies for American schools in all their 
diversity. 
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State ACCOUNTABiLnY study group members 

California Mcx law, 9045 Laguna Lake way. Elk Grove. CA 95624 

Rick Pntt, Legislative Analyst's OflQce. 925 "L" Street. Rm. IGOO. 
Sacramento. CA 95814. (916) 445-8641 

Colorado Jndy Bome*. supervisor of Planning and Evaluation Unit. Colorado 
Department of Education. 201 East Colfax. Denver. CO 8020G. (303) 
866-6824 

Ann KtMtser, Department of Planning and Evaluation. Colorado Springs 
Public Schools. 1 1 15 North El Paso St., Colorado Springs. CO 80903. (303) 
520-2000 

Connecticut Pat Focglone and Peter M. Ptowdft, State Department of Education. P.O. 
50x2219. Hartford. CT 06145. (203) 566-3577; 566-7117 

D'Ann Mazcocoi, Senior Research Analyst. OflQce of Legislative Research. 
18-20 Trinity. Hartford. CT 06106. (203) 566-8400 

Florida Thomas H. Fishei; Administrator. Department of Education. Assessment. 

Testing and Evaluation Section. Knott Bldg.. Rm. 506. Tallahassee. FL 
32399. (904) 488-8198 

Raj Tnniei; Assistant Superintendent of Accountability. Dade County 
Public Schools. 1^50 NE 2nd Ave.. Miami. FL 33132. (305) 376- 1506 

Georgia Joy Beny. Executive Director. Human Relations Commission of Georgia. 

270 Washington St.. SW. Room 603. Atlanta. GA 30334. (404) 651-91 15 

Lany Geaa, Director of Strategic Planning. Georgia State Department of 
Education. 1758 Twin Towers. Atlanta. GA 30334. (404) 656-6852 

Illinois Blaiy Loc Coi^lajaw, state Representative. 552 South Washington St.. 
Suite 119. Naplervllle. IL 60540. (217) 782-6507 

Gail Llebeiman, Assistant to the Governor for Education. Governor's Office. 
2-1/2 State House. Spr^igfleld. IL 62706. (217) 782-4921 

Salty Bnlkley Pancraaio, Chief. Research and Evaluation. Illinois State 
Board of Education. 100 North First Street. Springfield. IL 62777, (217) 
782-3950 
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MASSAC^^usETIS Alan Hartnum, Massachusetts Department Education, 1385 Hancock 

Street. Quliicy. MA 02 169. (617) 770-7204 

Rlduutd Knus, state Senator. Co-Chalr Education Committee. State 
House. Hoom 511. Boston. MA 02133. (617) 722-1432 

Robert Schwarts, Special Assistant to the Governor for Education. State 
House. Room 173. Boston. MA 02133. (617) 727-0770 

Minnesota David Bennett, superintendent of Schools. 360 Colbome. St. Paul. MN 
55102. (612) 293-5150 

Ken Kelsej, Chairperson. State Curriculum Advlsoiy Committee. 5835 
Sandy Drive. St. Cloud. MN 56301. (612) 255-3007 

James Pehlei; State Senate. Room 9. State Capitol. St. Paul. MN 55155. 
(612) 296-4241 

New jersey Valaxle French, Special Assistant to the Commissioner rf EducaUon, 

Department of Education. 225 W. State Street. CN 500. Trenton. NJ 08625. 
(609) 292 -4450 

Richaid Iflns, Commissioner of Education. Department of Education. 
MontpeUer. VT 05602. (802) 828-3135 

South Carolina Xeny Petenon, Wlnthrop Collie and Joint Business-Education legislative 

Committee. 1122 LacfySt.. Rm. 1005. Columbia. SC 29201. (803) 734-0489 

MOke Tnrner, Director. QflBce of School District. Accreditation and 
Assessment. SC State Department of Education. 701 Rutledge Building. 
Columbia. SC 29201. (803) 734-8536 

National governors* association 



Iflchael Coben, Nationa] Governors* Association. 444 North Capitol Street. 
Washington. DC 20202. (202) 624-7815 

Snaan Tiaiman, National Governors' Association. 444 North Capitol Street, 
Washington. DC 20202. (202) 624-5385 

Council of chief state school officers 

Ram^^ Selden, Council of Chief State Sch ^1 Officers. 400 North Capitol 
Street. Suite 377, Washington, DC 20001. (202) 624-7700 

National association of state boards of education 

Janice Eaile, National Association of State Boards of Education. 701 North 
Fairfiax Street, Suite 340. Alexandria, VA 22314, (703) 684-4000 
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National conference of state legislatures 



Marie Weston. National Conference of State L^lslatures. 150 17th Street. 
Suite 2100. Denver. CO 80263. (303) 623-7800 



Rand corporation 

Lomlne McDonnell and Jeannie Oakes. RAND Corporation. 1700 Main 
Street. Santa Monica. CA 90406. (2 13) 393-0411 

Office of educational research and improvement 

Emily O. Wnrtz. OERI. 555 New Jersey Ave.. NW. Washington. DC 20202. 
(202) 357-6172 
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Information sources 

For further infoimatlon about the design and use of educational indicators, 
we recommend the following publications: 

Jane L. Dsvld, 1987. Improvtng Educatton with LocaUy Developed 
Indicators. New Brunswick, NJ: Center for Poliqr Research in 
Education/Rutgers University. 

Richaxd J. Muniane and 8enU A. Ralzeut eds. 1988. Improvtng Indicators 
of the Quality of Scierux and MathenuitU^ 
Washington. DC: National Acaden^ Press. 

Jemnnle Oakes, 1986. Educational Indicators: A GiMt for Policyr^ 
Santa Monica. CA: Center for Policy Research in EducaOon/The RAND 
Corporatioa 

Representatives from each of the 10 States participating in the OERI Study 
Group prepared a detailed description of their States' accountability 
systems. Ihese reports can be obtained from: 

Emily O. WurtSt Office of Educational Research and Improvement. U.S. 
Department of Education. 555 New Jersey Ave.. NW. Washington. DC 20202 

The people and organizations listed below know and have experience with 
diffierent aspects of developing indicators and accountability systems. They 
may be contacted for further infomiation. 

Developing multiple indicators 

Eva Bakei; Center for Research on Evaluation. Standards, and Student 
Testing. University of California. Los Angeles. California 90024 

Jeannie Oakes, The RAND Corporation. 1700 Main St.. Santa Monica. 
California 90406 

Uartiii Oriand, Office of Educational Research and Improvement. U. S. 
Department of Education. 555 New Jersey Ave.. NW. Washington. DC 20202 

Senta A, Ralsen, Center for bnproving Science Education. 290 South Main 
St.. Andover. Massachusetts 01810 

Ramaqr Sekleiit Council of Chief State School Officers. Assessment Office, 
400 North Capitol Street. Suite 377. Washington. DC 20001 
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Alternatives to traditional standardized tests 



Joan Banm» Connecticut State Department of Education. P.O. Box 2219. 
Hartford. CT 06145 

Rexfoxd Brown, Education Commission of the States. 1860 Lincoln Street. 
Suite 300. Denver. CO 80295. 

Leigh Burtteln* Center for Research on Evaluation. Standards, and 
Student Testing. Unlveislty of California. Los Angeles. California 90024 

?U>bert CalfiM, Graduate School of Education. Stanford University. 
Stanford. California 94305 

Nomuui Fiedrlckfon* Director of Psychological Studies. Educational 
Testing Service. P.O. Box 2923. Princeton. NJ 08541 

umilam McMillan, Director of Assessment. Mlrmesota State Department of 
Education. Capitol Square Building. 550 Cedar St. St. Paul. Minnesota 
55101 

Ina MnlUSt National Assessment of Educational Progress. Educational 
Testing Service. P.O. Box 2923. Princeton. NJ 08541 

Implementing ACCOUNTABiLnY systems 

Micbael Cohen» National Governors' Association. HaU of the States. 444 
North Capitol Street Washington. DC 20001 

Ptt Forgtone, Connecticut State Department of Education. P.O. Box 2219. 
Hartford. CT 06145 

Lany Gcm, Director of Strategic Planning. Georgia State Department of 
Education. 1758 Twin Towers. Atlanta. GA 30334 

Ala Law, 9045 Laguna Lake Way. Elk Grove. CA 95624 

Lorraine McDonnell^ Hie RAND Corporation. 1700 Main Street. P.O. Box 
2138. Santa Monica. CA 90406 

Kamsay Selden. CoimcU of Chief State School QflOcers. Assessment OflKce. 
400 North Capitol Street. Suite 377. Washington. DC 20001 

Using assessment information for local school improvement 

Leigh Bunteln, Center for Research on Evaluation. Standards, and 
Student Testing. University of California. Los Angeles. California 90024 

Robert Calfee, Graduate School of Education. Stanford University. 
Stanford. California 94305 
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Jane L. David. 3144 David Avenue. Palo Alto. CA 94303 

Ken Kelaey. Chairperson. State Curriculum Advisory Committee. 5835 
Sandy Drive. St. Cloud. MN 56301 

Ann Kiaetser, Department of Planning and Evaluation. Colorado Spring 
Public Schools. 1115 North El Paso St.. Colorado Springs. CO 80903 

Andrew Portei; Institute for Research on Teaching. Michigan State 
University. Erlckson Hall. East Lansing. MI 48824 

Reporting ACCOUNTABiurY data 

Alex Law, 9045 Laguna Lake Way. Elk Grove. CA 95624 

John Hay, South Carolina Department of Education. 701 Rutledge 
Building. Columbia. SC 29201 

Sally Bnlklqr Pancrazio, Research and Evaluation. Illinois State 
Department of Education, 100 N. Front Street. Sprlngfleld. IL 62777 

Peter BL Prowda, Coraiectlcut State Department of Education, P.O. Box 
2219. Hartford. CT 06145 

Ramai^ Selden. Council of Chief State School Officers. Assessment Office. 
400 North Capitol Street. Suite 377. Washington, DC 20001 

Comparing school performance 

wmiam Codjr. CouncU of Chief State School Officers. Assessment Office. 
400 North Capitol Street. Suite 377. Washington, DC 20001 

Marie Fetler. Division of Program Evaluation and Research. California State 
Department of Education, 721 Capitol Mall. 4th Floor. Sacramento. CA 
95814 

Bfichael Tnmei; Accreditation and Assessment. South Carolina State 
Department of Education. 701 Rutledge Bulkilng. Columbia. SC 29201 

Ray Tnmer. Assistant Superintendent of Educational Accountability. Dade 
County Public Schools. 1450 NE 2nd Ave , Miami. FI, 33132 

Incentives, rewards, and sanctions 

David Bennett. St. Paul Schools. 360 Colbome. St. PauL MN 55102 

Saul Coopcnnan and Valarle French. New Jersey State Department of 
Education. 225 W. State Street. CN 500. Trentoa NJ 08625 
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John May, South Carolina Department of Education. 701 Rutledge 
Building. Columbia. SC 29201. 

National perspectives on state AccouNTABiLnY policies 

Michael Cohen and Susan Traiman, National Governors' As<Kx:latlon. Hall 
of the States. 444 North Capitol Street. Washington. DC 20001 

Terry Peterson, Wlnthrop College. 1 122 Lady St.. Room 1005. Columbia. 
SC 29101. (803) 734-0489 

Chris Plpho, Education Commission of the States. 1860 Lincoln Street. 
Denver. CO 80295 

Ramsay Selden, Council of Chief State School Ofikers. Assessment OfiQce. 
400 North Capitol Street. Suite 377. Washington. DC 20001 

BSark Weston, National Conference of State Legislatures. 150 17th Street. 
Suite 2100. Denver. CO 80265 
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Appendix C 



Status of council of chief state school officers 
educational indicators 



Tdde C.1-TOUCATIONAL INDICATORS ADDRESSING ODTCXMIES 



Indicators 



Mearaies 



Status 



Ttfget Date 



Attendance Rates 
Completion Rates 

Achievement Levels 



Percent Attend/Membership 

HS Graduates 
GEDs Awarded 

Perforaiance in Reading, Math, 
Science, Social Studies, and 
English at the Elementary, Inter- 
mediate and Secondary Levels 



Post-School Outcomes Job Placement 



Postsecondary Schooling 
Military 

Civic Involvement 

AtUtudes 

Student Followup 
Parents and Employers 



Under development 1988 

Recommended 1987 

Under development 1987 

Under development 1990 



To be developed 1989 

Available 1987 

Available 1987 

To be developed 1989 

To be developed 1989 

To be developed 1 989 
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nUe C.2-eDUCATI0NAL INDICArORS ADDRE88INO CQNISZT 



Indicators 


McsraxM 


Statu 


TaigetDate 


ocnool oysiciu 


1 ULcll OCilUUt~xl^C jrOylA UtUUIl 




lQft7 




PiUmOCX CUIU OiZC I\ailgCS» Ul 

School Districts 


JVfaJLUUJvZ 


190 # 




Percent School-Age Population 

T Trfistn /5^iiHi 11^2111 /Riirral 


Available 


1987 


Population 


Educational Attainment 


Available 


1987 






AVcUidQtC 








AVcUlf* PiC 




Resources 


Per Pupil Wealth 


Available 


1987 




Pre-School Participation 


To be developed 


1990 


Student Needs 


Percent Dlear^^antaged 


Available 


1987 




Percent Handicapped 


Recommended 


1988 




Percent Limited English 


Available 


1987 




Percent Private and Home 
Schooling 


To be developed 


1990 
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T«W€ C.3— INDICATORS ADDRESSING EDUCATIONAL POUOES AND PRACTICES 



Indicators 




OtfttUS 


Tuget Di 


INSTRUCnON 








uisuucuonai iimc 


Days/Years 


Under development 


1987 




Hours/Day 


Under dpvplonment 


1 QftT 




Pea:ent Engaged Time 


To be dpvploned 






Time Allocated To: Reading & 
Math. Science. Social Studies & 
Arts 


Under development 


1990 


Instructional Program 


Kindergarten Required 


Under development 


1987 




Graduation Requirements 


Under development 


1987 




Honors and AP Course Offerings 


Under development 


1988 


EflFective Schooling 


Instructional Leadership 


To be developed 


1987 




Instructional Environment 


To be developed 


1987 




Home and Community 
Involvement 


To be developed 


1987 


TEACHER QUAUIY 








Teacher Preparation 


Course Units 


Under development 


1987 




Testing 


Under development 


1987 




rractice leacning 


Under development 


1987 






Under development 


1987 


Teacher Work Force 


Percent Advanced Degrees 


Under development 


1988 




Experience In Years 


Under development 


1988 




Classroom I'eiiormance 


To be developed 


1990 


RESOURCE ALLOCATION 








Expenditures 


Total Per Pupil 


Under development 


1988 




Teacher Salaries 


Under development 


1988 




Instructional Materials 


Under development 


1988 


Effort 


Expenditures for Educ/Other 
Govt. Expenditures 


Available 


1987 


PAKnCIPAnON POLICIES 










Compulsory Age 


Under development 


1987 



6« 
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